public/rancher-docs
1
0
Fork 0
mirror of https://github.com/rancher/rancher-docs.git synced 2026-05-18 02:45:27 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #2953 from rancher/staging

Browse Source 
Merge staging (v2.5.4 docs) into master
This commit is contained in:
Catherine Luse
2021-01-07 11:55:24 -07:00
committed by GitHub
parent a8352914d2 b0914b21cf
commit 744c242bb8
59 changed files with 2922 additions and 1009 deletions
Download Patch File Download Diff File Expand all files Collapse all files
content/rancher/v2.x/en/backups/v2.0.x-v2.4.x/backup/docker-backups/_index.md
+14 -4
View File
@@ -12,17 +12,27 @@ aliases:
After completing your Docker installation of Rancher, we recommend creating backups of it on a regular basis. Having a recent backup will let you recover quickly from an unexpected disaster.
## Before You Start
### How to Read Placeholders
During the creation of your backup, you'll enter a series of commands, replacing placeholders with data from your environment. These placeholders are denoted with angled brackets and all capital letters (`<EXAMPLE>`). Here's an example of a command with a placeholder:
```
docker run --volumes-from rancher-data-<DATE> -v $PWD:/backup busybox tar pzcvf /backup/rancher-data-backup-<RANCHER_VERSION>-<DATE>.tar.gz /var/lib/rancher
docker run \
--volumes-from rancher-data-<DATE> \
-v $PWD:/backup busybox tar pzcvf /backup/rancher-data-backup-<RANCHER_VERSION>-<DATE>.tar.gz /var/lib/rancher
```
In this command, `<DATE>` is a placeholder for the date that the data container and backup were created. `9-27-18` for example.
Cross reference the image and reference table below to learn how to obtain this placeholder data. Write down or copy this information before starting the [procedure below](#creating-a-backup).
### Obtaining Placeholder Data
Get the placeholder data by running:
```
docker ps
```
Write down or copy this information before starting the [procedure below](#creating-a-backup).
<sup>Terminal `docker ps` Command, Displaying Where to Find `<RANCHER_CONTAINER_TAG>` and `<RANCHER_CONTAINER_NAME>`</sup>
![Placeholder Reference]({{<baseurl>}}/img/rancher/placeholder-ref.png)
@@ -37,7 +47,7 @@ Cross reference the image and reference table below to learn how to obtain this
You can obtain `<RANCHER_CONTAINER_TAG>` and `<RANCHER_CONTAINER_NAME>` by logging into your Rancher Server by remote connection and entering the command to view the containers that are running: `docker ps`. You can also view containers that are stopped with `docker ps -a`. Use these commands for help anytime while creating backups.
## Creating a Backup
### Creating a Backup
This procedure creates a backup that you can restore if Rancher encounters a disaster scenario.
content/rancher/v2.x/en/backups/v2.0.x-v2.4.x/backup/rke-backups/_index.md
+31 -17
View File
@@ -13,16 +13,24 @@ aliases:
---
This section describes how to create backups of your high-availability Rancher install.
>**Prerequisites:** {{< requirements_rollback >}}
## RKE Kubernetes Cluster Data
In an RKE installation, the cluster data is replicated on each of three etcd nodes in the cluster, providing redundancy and data duplication in case one of the nodes fails.
<figcaption>Architecture of an RKE Kubernetes Cluster Running the Rancher Management Server</figcaption>
<figcaption>Cluster Data within an RKE Kubernetes Cluster Running the Rancher Management Server</figcaption>
![Architecture of an RKE Kubernetes cluster running the Rancher management server]({{<baseurl>}}/img/rancher/rke-server-storage.svg)
## Backup Outline
# Requirements
### RKE Version
The commands for taking `etcd` snapshots are only available in RKE v0.1.7 and later.
### RKE Config File
You'll need the RKE config file that you used for Rancher install, `rancher-cluster.yml`. You created this file during your initial install. Place this file in same directory as the RKE binary.
# Backup Outline
Backing up your high-availability Rancher cluster is process that involves completing multiple tasks.
@@ -30,13 +38,12 @@ Backing up your high-availability Rancher cluster is process that involves compl
Take snapshots of your current `etcd` database using Rancher Kubernetes Engine (RKE).
1. [Store Snapshot(s) Externally](#2-backup-snapshots-to-a-safe-location)
1. [Store Snapshot(s) Externally](#2-back-up-local-snapshots-to-a-safe-location)
After taking your snapshots, export them to a safe location that won't be affected if your cluster encounters issues.
<br/>
### 1. Take Snapshots of the `etcd` Database
# 1. Take Snapshots of the `etcd` Database
Take snapshots of your `etcd` database. You can use these snapshots later to recover from a disaster scenario. There are two ways to take snapshots: recurringly, or as a one-off. Each option is better suited to a specific use case. Read the short description below each link to know when to use each option.
@@ -48,7 +55,7 @@ Take snapshots of your `etcd` database. You can use these snapshots later to rec
We advise taking one-time snapshots before events like upgrades or restoration of another snapshot.
#### Option A: Recurring Snapshots
### Option A: Recurring Snapshots
For all high-availability Rancher installs, we recommend taking recurring snapshots so that you always have a safe restoration point available.
@@ -103,7 +110,7 @@ To take recurring snapshots, enable the `etcd-snapshot` service, which is a serv
**Result:** RKE is configured to take recurring snapshots of `etcd` on all nodes running the `etcd` role. Snapshots are saved locally to the following directory: `/opt/rke/etcd-snapshots/`. If configured, the snapshots are also uploaded to your S3 compatible backend.
#### Option B: One-Time Snapshots
### Option B: One-Time Snapshots
When you're about to upgrade Rancher or restore it to a previous snapshot, you should snapshot your live image so that you have a backup of `etcd` in its last known state.
@@ -114,7 +121,9 @@ When you're about to upgrade Rancher or restore it to a previous snapshot, you s
2. Enter the following command. Replace `<SNAPSHOT.db>` with any name that you want to use for the snapshot (e.g. `upgrade.db`).
```
rke etcd snapshot-save --name <SNAPSHOT.db> --config rancher-cluster.yml
rke etcd snapshot-save \
--name <SNAPSHOT.db> \
--config rancher-cluster.yml
```
**Result:** RKE takes a snapshot of `etcd` running on each `etcd` node. The file is saved to `/opt/rke/etcd-snapshots`.
@@ -128,15 +137,20 @@ _Available as of RKE v0.2.0_
2. Enter the following command. Replace `<SNAPSHOT.db>` with any name that you want to use for the snapshot (e.g. `upgrade.db`).
```shell
rke etcd snapshot-save --config rancher-cluster.yml --name snapshot-name \
--s3 --access-key S3_ACCESS_KEY --secret-key S3_SECRET_KEY \
--bucket-name s3-bucket-name --s3-endpoint s3.amazonaws.com \
--folder folder-name # Available as of v2.3.0
rke etcd snapshot-save \
--config rancher-cluster.yml \
--name snapshot-name \
--s3 \
--access-key S3_ACCESS_KEY \
--secret-key S3_SECRET_KEY \
--bucket-name s3-bucket-name \
--s3-endpoint s3.amazonaws.com \
--folder folder-name # Available as of v2.3.0
```
**Result:** RKE takes a snapshot of `etcd` running on each `etcd` node. The file is saved to `/opt/rke/etcd-snapshots`. It is also uploaded to the S3 compatible backend.
### 2. Backup Local Snapshots to a Safe Location
# 2. Back up Local Snapshots to a Safe Location
> **Note:** If you are using RKE v0.2.0, you can enable saving the backups to a S3 compatible backend directly and skip this step.
content/rancher/v2.x/en/backups/v2.0.x-v2.4.x/restore/rke-restore/_index.md
+2
View File
@@ -34,6 +34,8 @@ Prepare by creating 3 new nodes to be the target for the restored Rancher instan
Alternatively you can re-use the existing nodes after clearing Kubernetes and Rancher configurations. This will destroy the data on these nodes. See [Node Cleanup]({{<baseurl>}}/rancher/v2.x/en/faq/cleaning-cluster-nodes/) for the procedure.
You must restore each of your etcd nodes to the same snapshot. Copy the snapshot you're using from one of your nodes to the others before running the `etcd snapshot-restore` command.
> **IMPORTANT:** Before starting the restore make sure all the Kubernetes services on the old cluster nodes are stopped. We recommend powering off the nodes to be sure.
### 2. Place Snapshot
content/rancher/v2.x/en/cis-scans/v2.5/_index.md
+217 -69
View File
@@ -4,51 +4,89 @@ shortTitle: Rancher v2.5
weight: 1
---
Rancher can run a security scan to check whether Kubernetes is deployed according to security best practices as defined in the CIS Kubernetes Benchmark.
The `rancher-cis-benchmark` app leverages <a href="https://github.com/aquasecurity/kube-bench" target="_blank">kube-bench,</a> an open-source tool from Aqua Security, to check clusters for CIS Kubernetes Benchmark compliance. Also, to generate a cluster-wide report, the application utilizes <a href="https://github.com/vmware-tanzu/sonobuoy" target="_blank">Sonobuoy</a> for report aggregation.
- [Changes in Rancher v2.5](#changes-in-rancher-v2-5)
- [About the CIS Benchmark](#about-the-cis-benchmark)
- [Installing rancher-cis-benchmark](#installing-rancher-cis-benchmark)
- [Uninstalling rancher-cis-benchmark](#uninstalling-rancher-cis-benchmark)
- [Running a Scan](#running-a-scan)
- [Skipping Tests](#skipping-tests)
- [Viewing Reports](#viewing-reports)
- [About the generated report](#about-the-generated-report)
- [About the Generated Report](#about-the-generated-report)
- [Test Profiles](#test-profiles)
- [About Skipped and Not Applicable Tests](#about-skipped-and-not-applicable-tests)
- [Roles-based access control](./rbac)
- [Roles-based Access Control](./rbac)
- [Configuration](./configuration)
- [How-to Guides](#how-to-guides)
- [Installing rancher-cis-benchmark](#installing-rancher-cis-benchmark)
- [Uninstalling rancher-cis-benchmark](#uninstalling-rancher-cis-benchmark)
- [Running a Scan](#running-a-scan)
- [Running a Scan Periodically on a Schedule](#running-a-scan-periodically-on-a-schedule)
- [Skipping Tests](#skipping-tests)
- [Viewing Reports](#viewing-reports)
- [Enabling Alerting for rancher-cis-benchmark](#enabling-alerting-for-rancher-cis-benchmark)
- [Configuring Alerts for a Periodic Scan on a Schedule](#configuring-alerts-for-a-periodic-scan-on-a-schedule)
- [Creating a Custom Benchmark Version for Running a Cluster Scan](#creating-a-custom-benchmark-version-for-running-a-cluster-scan)
### Changes in Rancher v2.5
# Changes in Rancher v2.5
We now support running CIS scans on any Kubernetes cluster, including hosted Kubernetes providers such as EKS, AKS, and GKE. Previously it was only supported to run CIS scans on RKE Kubernetes clusters.
In Rancher v2.4, the CIS scan tool was available from the **cluster manager** in the Rancher UI. Now it is available in the **Cluster Explorer** and it can be enabled and deployed using a Helm chart. It can be installed from the Rancher UI, but it can also be installed independently of Rancher. It deploys a CIS scan operator for the cluster, and deploys Kubernetes custom resources for cluster scans. The custom resources can be managed directly from the **Cluster Explorer.**
In v1 of the CIS scan tool, which was available in Rancher v2.4 through the cluster manager, recurring scans could be scheduled. The ability to schedule recurring scans is not yet available in Rancher v2.5.
In v1 of the CIS scan tool, which was available in Rancher v2.4 through the cluster manager, recurring scans could be scheduled. The ability to schedule recurring scans is now also available for CIS v2 from Rancher v2.5.4.
Support for alerting for the cluster scan results is not available for Rancher v2.5 yet.
Support for alerting for the cluster scan results is now also available from Rancher v2.5.4.
More test profiles were added. In Rancher v2.4, permissive and hardened profiles were included. In Rancher v2.5, the following profiles are available:
In Rancher v2.4, permissive and hardened profiles were included. In Rancher v2.5.0 and in v2.5.4, more profiles were included.
{{% tabs %}}
{{% tab "Profiles in v2.5.4" %}}
- Generic CIS 1.5
- Generic CIS 1.6
- RKE permissive 1.5
- RKE hardened 1.5
- RKE permissive 1.6
- RKE hardened 1.6
- EKS
- GKE
- RKE2 permissive 1.5
- RKE2 permissive 1.5
{{% /tab %}}
{{% tab "Profiles in v2.5.0-v2.5.3" %}}
- Generic CIS 1.5
- RKE permissive
- RKE hardened
- EKS
- GKE
{{% /tab %}}
{{% /tabs %}}
<br/>
The default profile depends on the type of cluster that will be scanned:
The default profile and the supported CIS benchmark version depends on the type of cluster that will be scanned and the Rancher version:
{{% tabs %}}
{{% tab "v2.5.4" %}}
The `rancher-cis-benchmark` supports the CIS 1.6 Benchmark version.
- For RKE Kubernetes clusters, the RKE Permissive 1.6 profile is the default.
- EKS and GKE have their own CIS Benchmarks published by `kube-bench`. The corresponding test profiles are used by default for those clusters.
- For RKE2 Kubernetes clusters, the RKE2 Permissive 1.5 profile is the default.
- For cluster types other than RKE, RKE2, EKS and GKE, the Generic CIS 1.5 profile will be used by default.
{{% /tab %}}
{{% tab "v2.5.0-v2.5.3" %}}
The `rancher-cis-benchmark` supports the CIS 1.5 Benchmark version.
- For RKE Kubernetes clusters, the RKE permissive profile is the default.
- EKS and GKE have their own CIS Benchmarks published by `kube-bench`. The corresponding test profiles are used by default for those clusters.
- For cluster types other than RKE, EKS and GKE, the Generic CIS 1.5 profile will be used by default.
The `rancher-cis-benchmark` currently supports the CIS 1.5 Benchmark version.
{{% /tab %}}
{{% /tabs %}}
> **Note:** CIS v1 cannot run on a cluster when CIS v2 is deployed. In other words, after `rancher-cis-benchmark` is installed, you can't run scans by going to the Cluster Manager view in the Rancher UI and clicking **Tools > CIS Scans.**
> **Note:** CIS v1 cannot run on a cluster when CIS v2 is deployed. In other words, after `rancher-cis-benchmark` is installed, you can't run scans by going to the Cluster Manager view in the Rancher UI and clicking <b>Tools > CIS Scans.</b>
# About the CIS Benchmark
@@ -59,7 +97,104 @@ CIS Benchmarks are best practices for the secure configuration of a target syste
The official Benchmark documents are available through the CIS website. The sign-up form to access the documents is
<a href="https://learn.cisecurity.org/benchmarks" target="_blank">here.</a>
# Installing rancher-cis-benchmark
# About the Generated Report
Each scan generates a report can be viewed in the Rancher UI and can be downloaded in CSV format.
From Rancher v2.5.4, the scan uses the CIS Benchmark v1.6 by default. In Rancher v2.5.0-2.5.3, the CIS Benchmark v1.5. is used.
The Benchmark version is included in the generated report.
The Benchmark provides recommendations of two types: Scored and Not Scored. Recommendations marked as Not Scored in the Benchmark are not included in the generated report.
Some tests are designated as "Not Applicable." These tests will not be run on any CIS scan because of the way that Rancher provisions RKE clusters. For information on how test results can be audited, and why some tests are designated to be not applicable, refer to Rancher's <a href="{{<baseurl>}}/rancher/v2.x/en/security/#the-cis-benchmark-and-self-assessment" target="_blank">self-assessment guide for the corresponding Kubernetes version.</a>
The report contains the following information:
| Column in Report | Description |
|------------------|-------------|
| `id` | The ID number of the CIS Benchmark. |
| `description` | The description of the CIS Benchmark test. |
| `remediation` | What needs to be fixed in order to pass the test. |
| `state` | Indicates if the test passed, failed, was skipped, or was not applicable. |
| `node_type` | The node role, which affects which tests are run on the node. Master tests are run on controlplane nodes, etcd tests are run on etcd nodes, and node tests are run on the worker nodes. |
| `audit` | This is the audit check that `kube-bench` runs for this test. |
| `audit_config` | Any configuration applicable to the audit script. |
| `test_info` | Test-related info as reported by `kube-bench`, if any. |
| `commands` | Test-related commands as reported by `kube-bench`, if any. |
| `config_commands` | Test-related configuration data as reported by `kube-bench`, if any. |
| `actual_value` | The test's actual value, present if reported by `kube-bench`. |
| `expected_result` | The test's expected result, present if reported by `kube-bench`. |
Refer to <a href="{{<baseurl>}}/rancher/v2.x/en/security/" target="_blank">the table in the cluster hardening guide</a> for information on which versions of Kubernetes, the Benchmark, Rancher, and our cluster hardening guide correspond to each other. Also refer to the hardening guide for configuration files of CIS-compliant clusters and information on remediating failed tests.
# Test Profiles
The following profiles are available:
{{% tabs %}}
{{% tab "Profiles in v2.5.4" %}}
- Generic CIS 1.5
- Generic CIS 1.6
- RKE permissive 1.5
- RKE hardened 1.5
- RKE permissive 1.6
- RKE hardened 1.6
- EKS
- GKE
- RKE2 permissive 1.5
- RKE2 permissive 1.5
{{% /tab %}}
{{% tab "Profiles in v2.5.0-v2.5.3" %}}
- Generic CIS 1.5
- RKE permissive
- RKE hardened
- EKS
- GKE
{{% /tab %}}
{{% /tabs %}}
You also have the ability to customize a profile by saving a set of tests to skip.
All profiles will have a set of not applicable tests that will be skipped during the CIS scan. These tests are not applicable based on how a RKE cluster manages Kubernetes.
There are two types of RKE cluster scan profiles:
- **Permissive:** This profile has a set of tests that have been will be skipped as these tests will fail on a default RKE Kubernetes cluster. Besides the list of skipped tests, the profile will also not run the not applicable tests.
- **Hardened:** This profile will not skip any tests, except for the non-applicable tests.
The EKS and GKE cluster scan profiles are based on CIS Benchmark versions that are specific to those types of clusters.
In order to pass the "Hardened" profile, you will need to follow the steps on the <a href="{{<baseurl>}}/rancher/v2.x/en/security/#rancher-hardening-guide" target="_blank">hardening guide</a> and use the `cluster.yml` defined in the hardening guide to provision a hardened cluster.
# About Skipped and Not Applicable Tests
For a list of skipped and not applicable tests, refer to <a href="{{<baseurl>}}/rancher/v2.x/en/cis-scans/skipped-tests" target="_blank">this page.</a>
For now, only user-defined skipped tests are marked as skipped in the generated report.
Any skipped tests that are defined as being skipped by one of the default profiles are marked as not applicable.
# Roles-based Access Control
For information about permissions, refer to <a href="{{<baseurl>}}/rancher/v2.x/en/cis-scans/rbac" target="_blank">this page.</a>
# Configuration
For more information about configuring the custom resources for the scans, profiles, and benchmark versions, refer to <a href="{{<baseurl>}}/rancher/v2.x/en/cis-scans/configuration" target="_blank">this page.</a>
# How-to Guides
- [Installing rancher-cis-benchmark](#installing-rancher-cis-benchmark)
- [Uninstalling rancher-cis-benchmark](#uninstalling-rancher-cis-benchmark)
- [Running a Scan](#running-a-scan)
- [Running a Scan Periodically on a Schedule](#running-a-scan-periodically-on-a-schedule)
- [Skipping Tests](#skipping-tests)
- [Viewing Reports](#viewing-reports)
- [Enabling Alerting for rancher-cis-benchmark](#enabling-alerting-for-rancher-cis-benchmark)
- [Configuring Alerts for a Periodic Scan on a Schedule](#configuring-alerts-for-a-periodic-scan-on-a-schedule)
- [Creating a Custom Benchmark Version for Running a Cluster Scan](#creating-a-custom-benchmark-version-for-running-a-cluster-scan)
### Installing rancher-cis-benchmark
1. In the Rancher UI, go to the **Cluster Explorer.**
1. Click **Apps.**
@@ -68,7 +203,7 @@ The official Benchmark documents are available through the CIS website. The sign
**Result:** The CIS scan application is deployed on the Kubernetes cluster.
# Uninstalling rancher-cis-benchmark
### Uninstalling rancher-cis-benchmark
1. From the **Cluster Explorer,** go to the top left dropdown menu and click **Apps & Marketplace.**
1. Click **Installed Apps.**
@@ -77,7 +212,7 @@ The official Benchmark documents are available through the CIS website. The sign
**Result:** The `rancher-cis-benchmark` application is uninstalled.
# Running a Scan
### Running a Scan
When a ClusterScan custom resource is created, it launches a new CIS scan on the cluster for the chosen ClusterScanProfile.
@@ -91,8 +226,26 @@ To run a scan,
1. Click **Create.**
**Result:** A report is generated with the scan results. To see the results, click the name of the scan that appears.
### Running a Scan Periodically on a Schedule
_Available as of v2.5.4_
# Skipping Tests
To run a ClusterScan on a schedule,
1. Go to the **Cluster Explorer** in the Rancher UI. In the top left dropdown menu, click **Cluster Explorer > CIS Benchmark.**
1. In the **Scans** section, click **Create.**
1. Choose a cluster scan profile. The profile determines which CIS Benchmark version will be used and which tests will be performed. If you choose the Default profile, then the CIS Operator will choose a profile applicable to the type of Kubernetes cluster it is installed on.
1. Choose the option **Run scan on a schedule.**
1. Enter a valid <a href="https://en.wikipedia.org/wiki/Cron#CRON_expression" target="_blank">cron schedule expression</a> in the field **Schedule.**
1. Choose a **Retention** count, which indicates the number of reports maintained for this recurring scan. By default this count is 3. When this retention limit is reached, older reports will get purged.
1. Click **Create.**
**Result:** The scan runs and reschedules to run according to the cron schedule provided. The **Next Scan** value indicates the next time this scan will run again.
A report is generated with the scan results every time the scan runs. To see the latest results, click the name of the scan that appears.
You can also see the previous reports by choosing the report from the **Reports** dropdown on the scan detail page.
### Skipping Tests
CIS scans can be run using test profiles with user-defined skips.
@@ -124,7 +277,7 @@ To skip tests, you will create a custom CIS scan profile. A profile contains the
When you [run a scan](#running-a-scan) that uses this profile, the defined tests will be skipped during the scan. The skipped tests will be marked in the generated report as `Skip`.
# Viewing Reports
### Viewing Reports
To view the generated CIS scan reports,
@@ -133,70 +286,65 @@ To view the generated CIS scan reports,
One can download the report from the Scans list or from the scan detail page.
# About the Generated Report
### Enabling Alerting for rancher-cis-benchmark
_Available as of v2.5.4_
Each scan generates a report can be viewed in the Rancher UI and can be downloaded in CSV format.
Alerts can be configured to be sent out for a scan that runs on a schedule.
In Rancher v2.5, the scan will use the CIS Benchmark v1.5. The Benchmark version is included in the generated report.
> **Prerequisite:**
>
> Before enabling alerts for `rancher-cis-benchmark`, make sure to install the `rancher-monitoring` application and configure the Receivers and Routes. For more information, see [this section.]({{<baseurl>}}/rancher/v2.x/en/monitoring-alerting/v2.5/configuration/alertmanager/)
>
> While configuring the routes for `rancher-cis-benchmark` alerts, you can specify the matching using the key-value pair `job: rancher-cis-scan`. An example route configuration is [here.]({{<baseurl>}}/rancher/v2.x/en/monitoring-alerting/v2.5/configuration/alertmanager/#example-route-config-for-cis-scan-alerts)
The Benchmark provides recommendations of two types: Scored and Not Scored. Recommendations marked as Not Scored in the Benchmark are not included in the generated report.
While installing or upgrading the `rancher-cis-benchmark` application, set the following flag to `true` in the `values.yaml`:
Some tests are designated as "Not Applicable." These tests will not be run on any CIS scan because of the way that Rancher provisions RKE clusters. For information on how test results can be audited, and why some tests are designated to be not applicable, refer to Rancher's <a href="{{<baseurl>}}/rancher/v2.x/en/security/#the-cis-benchmark-and-self-assessment" target="_blank">self-assessment guide for the corresponding Kubernetes version.</a>
```yaml
alerts:
enabled: true
```
The report contains the following information:
### Configuring Alerts for a Periodic Scan on a Schedule
_Available as of v2.5.4_
| Column in Report | Description |
|------------------|-------------|
| `id` | The ID number of the CIS Benchmark. |
| `description` | The description of the CIS Benchmark test. |
| `remediation` | What needs to be fixed in order to pass the test. |
| `state` | Indicates if the test passed, failed, was skipped, or was not applicable. |
| `node_type` | The node role, which affects which tests are run on the node. Master tests are run on controlplane nodes, etcd tests are run on etcd nodes, and node tests are run on the worker nodes. |
| `audit` | This is the audit check that `kube-bench` runs for this test. |
| `audit_config` | Any configuration applicable to the audit script. |
| `test_info` | Test-related info as reported by `kube-bench`, if any. |
| `commands` | Test-related commands as reported by `kube-bench`, if any. |
| `config_commands` | Test-related configuration data as reported by `kube-bench`, if any. |
| `actual_value` | The test's actual value, present if reported by `kube-bench`. |
| `expected_result` | The test's expected result, present if reported by `kube-bench`. |
From Rancher v2.5.4, it is possible to run a ClusterScan on a schedule.
Refer to <a href="{{<baseurl>}}/rancher/v2.x/en/security/" target="_blank">the table in the cluster hardening guide</a> for information on which versions of Kubernetes, the Benchmark, Rancher, and our cluster hardening guide correspond to each other. Also refer to the hardening guide for configuration files of CIS-compliant clusters and information on remediating failed tests.
A scheduled scan can also specify if you should receive alerts when the scan completes.
# Test Profiles
Alerts are supported only for a scan that runs on a schedule.
The following profiles are available:
The `rancher-cis-benchmark` application supports two types of alerts:
- Generic CIS 1.5 (default)
- RKE permissive
- RKE hardened
- EKS
- GKE
- Alert on scan completion: This alert is sent out when the scan run finishes. The alert includes details including the ClusterScan's name and the ClusterScanProfile name.
- Alert on scan failure: This alert is sent out if there are some test failures in the scan run or if the scan is in a `Fail` state.
You also have the ability to customize a profile by saving a set of tests to skip.
> **Prerequisite:**
>
> Before enabling alerts for `rancher-cis-benchmark`, make sure to install the `rancher-monitoring` application and configure the Receivers and Routes. For more information, see [this section.]({{<baseurl>}}/rancher/v2.x/en/monitoring-alerting/v2.5/configuration/alertmanager/)
>
> While configuring the routes for `rancher-cis-benchmark` alerts, you can specify the matching using the key-value pair `job: rancher-cis-scan`. An example route configuration is [here.]({{<baseurl>}}/rancher/v2.x/en/monitoring-alerting/v2.5/configuration/alertmanager/#example-route-config-for-cis-scan-alerts)
All profiles will have a set of not applicable tests that will be skipped during the CIS scan. These tests are not applicable based on how a RKE cluster manages Kubernetes.
To configure alerts for a scan that runs on a schedule,
There are 2 types of RKE cluster scan profiles:
1. Please enable alerts on the `rancher-cis-benchmark` application (#enabling-alerting-for-rancher-cis-benchmark)
1. Go to the **Cluster Explorer** in the Rancher UI. In the top left dropdown menu, click **Cluster Explorer > CIS Benchmark.**
1. In the **Scans** section, click **Create.**
1. Choose a cluster scan profile. The profile determines which CIS Benchmark version will be used and which tests will be performed. If you choose the Default profile, then the CIS Operator will choose a profile applicable to the type of Kubernetes cluster it is installed on.
1. Choose the option **Run scan on a schedule.**
1. Enter a valid [cron schedule expression](https://en.wikipedia.org/wiki/Cron#CRON_expression) in the field **Schedule.**
1. Check the boxes next to the Alert types under **Alerting.**
1. Optional: Choose a **Retention** count, which indicates the number of reports maintained for this recurring scan. By default this count is 3. When this retention limit is reached, older reports will get purged.
1. Click **Create.**
- **Permissive:** This profile has a set of tests that have been will be skipped as these tests will fail on a default RKE Kubernetes cluster. Besides the list of skipped tests, the profile will also not run the not applicable tests.
- **Hardened:** This profile will not skip any tests, except for the non-applicable tests.
**Result:** The scan runs and reschedules to run according to the cron schedule provided. Alerts are sent out when the scan finishes if routes and receiver are configured under `rancher-monitoring` application.
The EKS and GKE cluster scan profiles are based on CIS Benchmark versions that are specific to those types of clusters.
A report is generated with the scan results every time the scan runs. To see the latest results, click the name of the scan that appears.
In order to pass the "Hardened" profile, you will need to follow the steps on the <a href="{{<baseurl>}}/rancher/v2.x/en/security/#rancher-hardening-guide" target="_blank">hardening guide</a> and use the `cluster.yml` defined in the hardening guide to provision a hardened cluster.
### Creating a Custom Benchmark Version for Running a Cluster Scan
_Available as of v2.5.4_
There could be some Kubernetes cluster setups that require custom configurations of the Benchmark tests. For example, the path to the Kubernetes config files or certs might be different than the standard location where the upstream CIS Benchmarks look for them.
# About Skipped and Not Applicable Tests
It is now possible to create a custom Benchmark Version for running a cluster scan using the `rancher-cis-benchmark` application.
For a list of skipped and not applicable tests, refer to <a href="{{<baseurl>}}/rancher/v2.x/en/cis-scans/skipped-tests" target="_blank">this page.</a>
For now, only user-defined skipped tests are marked as skipped in the generated report.
Any skipped tests that are defined as being skipped by one of the default profiles are marked as not applicable.
# Roles-based Access Control
For information about permissions, refer to <a href="{{<baseurl>}}/rancher/v2.x/en/cis-scans/rbac" target="_blank">this page.</a>
# Configuration
For more information about configuring the custom resources for the scans, profiles, and benchmark versions, refer to <a href="{{<baseurl>}}/rancher/v2.x/en/cis-scans/configuration" target="_blank">this page.</a>
For details, see [this page.](./custom-benchmark)
content/rancher/v2.x/en/cis-scans/v2.5/custom-benchmark/_index.md
+82
View File
@@ -0,0 +1,82 @@
---
title: Creating a Custom Benchmark Version for Running a Cluster Scan
weight: 4
---
_Available as of v2.5.4_
Each Benchmark Version defines a set of test configuration files that define the CIS tests to be run by the <a href="https://github.com/aquasecurity/kube-bench" target="_blank">kube-bench</a> tool.
The `rancher-cis-benchmark` application installs a few default Benchmark Versions which are listed under CIS Benchmark application menu.
But there could be some Kubernetes cluster setups that require custom configurations of the Benchmark tests. For example, the path to the Kubernetes config files or certs might be different than the standard location where the upstream CIS Benchmarks look for them.
It is now possible to create a custom Benchmark Version for running a cluster scan using the `rancher-cis-benchmark` application.
When a cluster scan is run, you need to select a Profile which points to a specific Benchmark Version.
Follow all the steps below to add a custom Benchmark Version and run a scan using it.
1. [Prepare the Custom Benchmark Version ConfigMap](#1-prepare-the-custom-benchmark-version-configmap)
2. [Add a Custom Benchmark Version to a Cluster](#2-add-a-custom-benchmark-version-to-a-cluster)
3. [Create a New Profile for the Custom Benchmark Version](#3-create-a-new-profile-for-the-custom-benchmark-version)
4. [Run a Scan Using the Custom Benchmark Version](#4-run-a-scan-using-the-custom-benchmark-version)
### 1. Prepare the Custom Benchmark Version ConfigMap
To create a custom benchmark version, first you need to create a ConfigMap containing the benchmark version's config files and upload it to your Kubernetes cluster where you want to run the scan.
To prepare a custom benchmark version ConfigMap, suppose we want to add a custom Benchmark Version named `foo`.
1. Create a directory named `foo` and inside this directory, place all the config YAML files that the <a href="https://github.com/aquasecurity/kube-bench" target="_blank">kube-bench</a> tool looks for. For example, here are the config YAML files for a Generic CIS 1.5 Benchmark Version https://github.com/aquasecurity/kube-bench/tree/master/cfg/cis-1.5
1. Place the complete `config.yaml` file, which includes all the components that should be tested.
1. Add the Benchmark version name to the `target_mapping` section of the `config.yaml`:
```yaml
target_mapping:
"foo":
- "master"
- "node"
- "controlplane"
- "etcd"
- "policies"
```
1. Upload this directory to your Kubernetes Cluster by creating a ConfigMap:
```yaml
kubectl create configmap -n <namespace> foo --from-file=<path to directory foo>
```
### 2. Add a Custom Benchmark Version to a Cluster
1. Once the ConfigMap has been created in your cluster, navigate to the **Cluster Explorer** in the Rancher UI.
1. In the top left dropdown menu, click **Cluster Explorer > CIS Benchmark.**
1. In the **Benchmark Versions** section, click **Create.**
1. Enter the **Name** and a description for your custom benchmark version.
1. Choose the cluster provider that your benchmark version applies to.
1. Choose the ConfigMap you have uploaded from the dropdown.
1. Add the minimum and maximum Kubernetes version limits applicable, if any.
1. Click **Create.**
### 3. Create a New Profile for the Custom Benchmark Version
To run a scan using your custom benchmark version, you need to add a new Profile pointing to this benchmark version.
1. Once the custom benchmark version has been created in your cluster, navigate to the **Cluster Explorer** in the Rancher UI.
1. In the top left dropdown menu, click **Cluster Explorer > CIS Benchmark.**
1. In the **Profiles** section, click **Create.**
1. Provide a **Name** and description. In this example, we name it `foo-profile`.
1. Choose the Benchmark Version `foo` from the dropdown.
1. Click **Create.**
### 4. Run a Scan Using the Custom Benchmark Version
Once the Profile pointing to your custom benchmark version `foo` has been created, you can create a new Scan to run the custom test configs in the Benchmark Version.
To run a scan,
1. Go to the **Cluster Explorer** in the Rancher UI. In the top left dropdown menu, click **Cluster Explorer > CIS Benchmark.**
1. In the **Scans** section, click **Create.**
1. Choose the new cluster scan profile `foo-profile`.
1. Click **Create.**
**Result:** A report is generated with the scan results. To see the results, click the name of the scan that appears.
content/rancher/v2.x/en/cis-scans/v2.5/rbac/_index.md
+1
View File
@@ -11,6 +11,7 @@ This section describes the permissions required to use the rancher-cis-benchmark
The rancher-cis-benchmark is a cluster-admin only feature by default.
However, the `rancher-cis-benchmark` chart installs these two default `ClusterRoles`:
- cis-admin
- cis-view
content/rancher/v2.x/en/cluster-provisioning/hosted-kubernetes-clusters/eks/_index.md
+9 -1
View File
@@ -135,6 +135,14 @@ Optional: To encrypt secrets, select or enter a key created in [AWS Key Manageme
Configuring Public/Private API access is an advanced use case. For details, refer to the EKS cluster endpoint access control [documentation.](https://docs.aws.amazon.com/eks/latest/userguide/cluster-endpoint.html)
### Private-only API Endpoints
If you enable private and disable public API endpoint access when creating a cluster, then there is an extra step you must take in order for Rancher to connect to the cluster successfully. In this case, a pop-up will be displayed with a command that you will run on the cluster to register it with Rancher. Once the cluster is provisioned, you can run the displayed command anywhere you can connect to the cluster's Kubernetes API.
There are two ways to avoid this extra manual step:
* You can create the cluster with both private and public API endpoint access on cluster creation. You can disable public access after the cluster is created and in an active state and Rancher will continue to communicate with the EKS cluster.
* You can ensure that Rancher shares a subnet with the EKS cluster. Then security groups can be used to enable Rancher to communicate with the cluster's API endpoint. In this case, the command to register the cluster is not needed, and Rancher will be able to communicate with your cluster. For more information on configuring security groups, refer to the [security groups documentation](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_SecurityGroups.html).
### Public Access Endpoints
<a id="public-access-endpoints-2-5"></a>
@@ -564,7 +572,7 @@ UpstreamSpec represents the cluster as it is in EKS and is refreshed on an inter
The effective desired state can be thought of as the UpstreamSpec + all non-nil fields in the EKSConfig. This is what is displayed in the UI.
If Rancher and another source attempt to update an EKS cluster at the same time or within the 5 minute refresh window of an update finishing, then it is likely any “managed” fields can be caught in a race condition. For example, a cluster may have PrivateAccess as a managed field. If PrivateAccess is false and then enabled in EKS console, then finishes at 11:01, and then tags are updated from Rancher before 11:05 the value will likely be overwritten. This would also occur if tags were updated while the cluster was processing the update. If the cluster was registered and the PrivateAccess fields was nil then this issue should not occur in the aforementioend case.
If Rancher and another source attempt to update an EKS cluster at the same time or within the 5 minute refresh window of an update finishing, then it is likely any “managed” fields can be caught in a race condition. For example, a cluster may have PrivateAccess as a managed field. If PrivateAccess is false and then enabled in EKS console, then finishes at 11:01, and then tags are updated from Rancher before 11:05 the value will likely be overwritten. This would also occur if tags were updated while the cluster was processing the update. If the cluster was registered and the PrivateAccess fields was nil then this issue should not occur in the aforementioned case.
### Configuring the Refresh Interval
content/rancher/v2.x/en/cluster-provisioning/hosted-kubernetes-clusters/gke/_index.md
+5 -1
View File
@@ -38,6 +38,10 @@ Use {{< product >}} to set up and configure your Kubernetes cluster.
>**Note:** After submitting your private key, you may have to enable the Google Kubernetes Engine API. If prompted, browse to the URL displayed in the Rancher UI to enable the API.
6. Select your **Cluster Options**, customize your **Nodes** and customize the **Security** for the GKE cluster. Review your options to confirm they're correct. Then click **Create**.
6. Select your **Cluster Options**
7. Customize your **Node Options**
* Enabling the Auto Upgrade feature for Nodes is not recommended.
8. Select your **Security Options**
9. Review your options to confirm they're correct. Then click **Create**.
{{< result_create-cluster >}}
content/rancher/v2.x/en/cluster-provisioning/node-requirements/_index.md
+58 -125
View File
@@ -1,13 +1,11 @@
---
title: Node Requirements for User Clusters
title: Node Requirements for Rancher Managed Clusters
weight: 1
---
This page describes the requirements for the nodes where your apps and services will be installed.
This page describes the requirements for the Rancher managed Kubernetes clusters where your apps and services will be installed. These downstream clusters should be separate from the cluster (or single node) running Rancher.
In this section, "user cluster" refers to a cluster running your apps, which should be separate from the cluster (or single node) running Rancher.
> If Rancher is installed on a high-availability Kubernetes cluster, the Rancher server cluster and user clusters have different requirements. For Rancher installation requirements, refer to the node requirements in the [installation section.]({{<baseurl>}}/rancher/v2.x/en/installation/requirements/)
> If Rancher is installed on a high-availability Kubernetes cluster, the Rancher server cluster and downstream clusters have different requirements. For Rancher installation requirements, refer to the node requirements in the [installation section.]({{<baseurl>}}/rancher/v2.x/en/installation/requirements/)
Make sure the nodes for the Rancher server fulfill the following requirements:
@@ -28,11 +26,63 @@ If you plan to use ARM64, see [Running on ARM64 (Experimental).]({{<baseurl>}}/r
For information on how to install Docker, refer to the official [Docker documentation.](https://docs.docker.com/)
### Oracle Linux and RHEL Derived Linux Nodes
Some distributions of Linux derived from RHEL, including Oracle Linux, may have default firewall rules that block communication with Helm. We recommend disabling firewalld. For Kubernetes 1.19, firewalld must be turned off.
SUSE Linux may have a firewall that blocks all ports by default. In that situation, follow [these steps](#opening-suse-linux-ports) to open the ports needed for adding a host to a custom cluster.
### SUSE Linux Nodes
### Requirements for Windows Nodes
SUSE Linux may have a firewall that blocks all ports by default. In that situation, follow [these steps]({{<baseurl>}}/rancher/v2.x/en/installation/requirements/ports/#opening-suse-linux-ports) to open the ports needed for adding a host to a custom cluster.
### Flatcar Container Linux Nodes
When [Launching Kubernetes with Rancher]({{<baseurl>}}/rancher/v2.x/en/cluster-provisioning/rke-clusters/) using Flatcar Container Linux nodes, it is required to use the following configuration in the [Cluster Config File]({{<baseurl>}}/rancher/v2.x/en/cluster-provisioning/rke-clusters/options/#cluster-config-file)
{{% tabs %}}
{{% tab "Canal"%}}
```yaml
rancher_kubernetes_engine_config:
network:
plugin: canal
options:
canal_flex_volume_plugin_dir: /opt/kubernetes/kubelet-plugins/volume/exec/nodeagent~uds
flannel_backend_type: vxlan
services:
kube-controller:
extra_args:
flex-volume-plugin-dir: /opt/kubernetes/kubelet-plugins/volume/exec/
```
{{% /tab %}}
{{% tab "Calico"%}}
```yaml
rancher_kubernetes_engine_config:
network:
plugin: calico
options:
calico_flex_volume_plugin_dir: /opt/kubernetes/kubelet-plugins/volume/exec/nodeagent~uds
flannel_backend_type: vxlan
services:
kube-controller:
extra_args:
flex-volume-plugin-dir: /opt/kubernetes/kubelet-plugins/volume/exec/
```
{{% /tab %}}
{{% /tabs %}}
It is also required to enable the Docker service, you can enable the Docker service using the following command:
```
systemctl enable docker.service
```
The Docker service is enabled automatically when using [Node Drivers]({{<baseurl>}}/rancher/v2.x/en/admin-settings/drivers/#node-drivers).
### Windows Nodes
_Windows worker nodes can be used as of Rancher v2.3.0_
@@ -58,127 +108,10 @@ The ports required to be open are different depending on how the user cluster is
For a breakdown of the port requirements for etcd nodes, controlplane nodes, and worker nodes in a Kubernetes cluster, refer to the [port requirements for the Rancher Kubernetes Engine.]({{<baseurl>}}/rke/latest/en/os/#ports)
Details on which ports are used in each situation are found in the following sections:
- [Commonly used ports](#commonly-used-ports)
- [Port requirements for custom clusters](#port-requirements-for-custom-clusters)
- [Port requirements for clusters hosted by an infrastructure provider](#port-requirements-for-clusters-hosted-by-an-infrastructure-provider)
- [Security group for nodes on AWS EC2](#security-group-for-nodes-on-aws-ec2)
- [Port requirements for clusters hosted by a Kubernetes provider](#port-requirements-for-clusters-hosted-by-a-kubernetes-provider)
- [Port requirements for imported clusters](#port-requirements-for-imported-clusters)
- [Port requirements for local traffic](#port-requirements-for-local-traffic)
### Commonly Used Ports
If security isn't a large concern and you're okay with opening a few additional ports, you can use this table as your port reference instead of the comprehensive tables in the following sections.
These ports are typically opened on your Kubernetes nodes, regardless of what type of cluster it is.
<figcaption>Commonly Used Ports Reference</figcaption>
| Protocol | Port | Description |
|:--------: |:----------------: |------------------------------------------------- |
| TCP | 22 | Node driver SSH provisioning |
| TCP | 2376 | Node driver Docker daemon TLS port |
| TCP | 2379 | etcd client requests |
| TCP | 2380 | etcd peer communication |
| UDP | 8472 | Canal/Flannel VXLAN overlay networking |
| UDP | 4789 | Flannel VXLAN overlay networking on Windows cluster |
| TCP | 9099 | Canal/Flannel livenessProbe/readinessProbe |
| TCP | 6783 | Weave Port |
| UDP | 6783-6784 | Weave UDP Ports |
| TCP | 10250 | kubelet API |
| TCP | 10254 | Ingress controller livenessProbe/readinessProbe |
| TCP/UDP | 30000-</br>32767 | NodePort port range |
### Port Requirements for Custom Clusters
If you are launching a Kubernetes cluster on your existing infrastructure, refer to these port requirements.
The following table depicts the port requirements for [Rancher Launched Kubernetes]({{<baseurl>}}/rancher/v2.x/en/cluster-provisioning/rke-clusters/) with [custom nodes]({{<baseurl>}}/rancher/v2.x/en/cluster-provisioning/rke-clusters/custom-nodes/).
{{< ports-custom-nodes >}}
### Port Requirements for Clusters Hosted by an Infrastructure Provider
If you are launching a Kubernetes cluster on nodes that are in an infrastructure provider such as Amazon EC2, Google Container Engine, DigitalOcean, Azure, or vSphere, these port requirements apply.
These required ports are automatically opened by Rancher during creation of clusters using cloud providers.
The following table depicts the port requirements for [Rancher Launched Kubernetes]({{<baseurl>}}/rancher/v2.x/en/cluster-provisioning/rke-clusters/) with nodes created in an [Infrastructure Provider]({{<baseurl>}}/rancher/v2.x/en/cluster-provisioning/rke-clusters/node-pools/).
>**Note:**
>The required ports are automatically opened by Rancher during creation of clusters in cloud providers like Amazon EC2 or DigitalOcean.
{{< ports-iaas-nodes >}}
#### Security Group for Nodes on AWS EC2
When using the [AWS EC2 node driver]({{<baseurl>}}/rancher/v2.x/en/cluster-provisioning/rke-clusters/node-pools/ec2/) to provision cluster nodes in Rancher, you can choose to let Rancher create a security group called `rancher-nodes`. The following rules are automatically added to this security group.
| Type | Protocol | Port Range | Source/Destination | Rule Type |
|-----------------|:--------:|:-----------:|------------------------|:---------:|
| SSH | TCP | 22 | 0.0.0.0/0 | Inbound |
| HTTP | TCP | 80 | 0.0.0.0/0 | Inbound |
| Custom TCP Rule | TCP | 443 | 0.0.0.0/0 | Inbound |
| Custom TCP Rule | TCP | 2376 | 0.0.0.0/0 | Inbound |
| Custom TCP Rule | TCP | 2379-2380 | sg-xxx (rancher-nodes) | Inbound |
| Custom UDP Rule | UDP | 4789 | sg-xxx (rancher-nodes) | Inbound |
| Custom TCP Rule | TCP | 6443 | 0.0.0.0/0 | Inbound |
| Custom UDP Rule | UDP | 8472 | sg-xxx (rancher-nodes) | Inbound |
| Custom TCP Rule | TCP | 10250-10252 | sg-xxx (rancher-nodes) | Inbound |
| Custom TCP Rule | TCP | 10256 | sg-xxx (rancher-nodes) | Inbound |
| Custom TCP Rule | TCP | 30000-32767 | 0.0.0.0/0 | Inbound |
| Custom UDP Rule | UDP | 30000-32767 | 0.0.0.0/0 | Inbound |
| All traffic | All | All | 0.0.0.0/0 | Outbound |
### Port Requirements for Clusters Hosted by a Kubernetes Provider
If you are launching a cluster with a [hosted Kubernetes provider]({{<baseurl>}}/rancher/v2.x/en/cluster-provisioning/hosted-kubernetes-clusters). such as Google Kubernetes Engine, Amazon EKS, or Azure Kubernetes Service, refer to these port requirements.
{{< ports-imported-hosted >}}
### Port Requirements for Imported Clusters
The following table depicts the port requirements for [imported clusters]({{<baseurl>}}/rancher/v2.x/en/cluster-provisioning/imported-clusters/).
{{< ports-imported-hosted >}}
### Port Requirements for Local Traffic
Ports marked as `local traffic` (i.e., `9099 TCP`) in the port requirements are used for Kubernetes healthchecks (`livenessProbe` and`readinessProbe`).
These healthchecks are executed on the node itself. In most cloud environments, this local traffic is allowed by default.
However, this traffic may be blocked when:
- You have applied strict host firewall policies on the node.
- You are using nodes that have multiple interfaces (multihomed).
In these cases, you have to explicitly allow this traffic in your host firewall, or in case of public/private cloud hosted machines (i.e. AWS or OpenStack), in your security group configuration. Keep in mind that when using a security group as source or destination in your security group, explicitly opening ports only applies to the private interface of the nodes/instances.
Details on which ports are used in each situation are found under [Downstream Cluster Port Requirements](({{<baseurl>}}/rancher/v2.x/en/installation/requirements/ports#downstream-kubernetes-cluster-nodes)).
# Optional: Security Considerations
If you want to provision a Kubernetes cluster that is compliant with the CIS (Center for Internet Security) Kubernetes Benchmark, we recommend to following our hardening guide to configure your nodes before installing Kubernetes.
For more information on the hardening guide and details on which version of the guide corresponds to your Rancher and Kubernetes versions, refer to the [security section.]({{<baseurl>}}/rancher/v2.x/en/security/#rancher-hardening-guide)
# Opening SUSE Linux Ports
SUSE Linux may have a firewall that blocks all ports by default. To open the ports needed for adding the host to a custom cluster,
1. SSH into the instance.
1. Edit /`etc/sysconfig/SuSEfirewall2` and open the required ports. In this example, ports 9796 and 10250 are also opened for monitoring:
```
FW_SERVICES_EXT_TCP="22 80 443 2376 2379 2380 6443 9099 9796 10250 10254 30000:32767"
FW_SERVICES_EXT_UDP="8472 30000:32767"
FW_ROUTE=yes
```
1. Restart the firewall with the new ports:
```
SuSEfirewall2
```
**Result:** The node has the open ports required to be added to a custom cluster.
content/rancher/v2.x/en/cluster-provisioning/rke-clusters/rancher-agents/_index.md
+25 -3
View File
@@ -22,16 +22,38 @@ The `cattle-node-agent` is used to interact with nodes in a [Rancher Launched Ku
### Scheduling rules
_Applies to v2.3.0 and higher_
_Applies to v2.5.4 and higher_
Starting with Rancher v2.5.4, the tolerations for the `cattle-cluster-agent` changed from `operator:Exists` (allowing all taints) to a fixed set of tolerations (listed below, if no controlplane nodes are visible in the cluster) or dynamically added tolerations based on taints applied to the controlplane nodes. This change was made to allow [Taint based Evictions](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/#taint-based-evictions) to work properly for `cattle-cluster-agent`. The default tolerations are described below. If controlplane nodes are present the cluster, the tolerations will be replaced with tolerations matching the taints on the controlplane nodes.
| Component | nodeAffinity nodeSelectorTerms | nodeSelector | Tolerations |
| ---------------------- | ------------------------------------------ | ------------ | ------------------------------------------------------------------------------ |
| `cattle-cluster-agent` | `beta.kubernetes.io/os:NotIn:windows` | none | **Note:** These are the default tolerations, and will be replaced by tolerations matching taints applied to controlplane nodes.<br/><br/>`effect:NoSchedule`<br/>`key:node-role.kubernetes.io/controlplane`<br/>`value:true`<br/><br/>`effect:NoSchedule`<br/>`key:node-role.kubernetes.io/control-plane`<br/>`operator:Exists`<br/><br/>`effect:NoSchedule`<br/>`key:node-role.kubernetes.io/master`<br/>`operator:Exists` |
| `cattle-node-agent` | `beta.kubernetes.io/os:NotIn:windows` | none | `operator:Exists` |
The `cattle-cluster-agent` Deployment has preferred scheduling rules using `preferredDuringSchedulingIgnoredDuringExecution`, favoring to be scheduled on nodes with the `controlplane` node. When there are no controlplane nodes visible in the cluster (this is usually the case when using [Clusters from Hosted Kubernetes Providers]({{<baseurl>}}/rancher/v2.x/en/cluster-provisioning/hosted-kubernetes-clusters/)), you can add the label `cattle.io/cluster-agent=true` on a node to prefer scheduling the `cattle-cluster-agent` pod to that node.
See [Kubernetes: Assigning Pods to Nodes](https://kubernetes.io/docs/concepts/configuration/assign-pod-node/) to find more information about scheduling rules.
The `preferredDuringSchedulingIgnoredDuringExecution` configuration is shown in the table below:
| Weight | Expression |
| ------ | ------------------------------------------------ |
| 100 | `node-role.kubernetes.io/controlplane:In:"true"` |
| 100 | `node-role.kubernetes.io/control-plane:In:"true"` |
| 100 | `node-role.kubernetes.io/master:In:"true"` |
| 1 | `cattle.io/cluster-agent:In:"true"` |
_Applies to v2.3.0 up to v2.5.3_
| Component | nodeAffinity nodeSelectorTerms | nodeSelector | Tolerations |
| ---------------------- | ------------------------------------------ | ------------ | ------------------------------------------------------------------------------ |
| `cattle-cluster-agent` | `beta.kubernetes.io/os:NotIn:windows` | none | `operator:Exists` |
| `cattle-node-agent` | `beta.kubernetes.io/os:NotIn:windows` | none | `operator:Exists` |
The `cattle-cluster-agent` Deployment has preferred scheduling rules using `requiredDuringSchedulingIgnoredDuringExecution`, favoring to be scheduled on nodes with the `controlplane` node. See [Kubernetes: Assigning Pods to Nodes](https://kubernetes.io/docs/concepts/configuration/assign-pod-node/) to find more information about scheduling rules.
The `cattle-cluster-agent` Deployment has preferred scheduling rules using `preferredDuringSchedulingIgnoredDuringExecution`, favoring to be scheduled on nodes with the `controlplane` node. See [Kubernetes: Assigning Pods to Nodes](https://kubernetes.io/docs/concepts/configuration/assign-pod-node/) to find more information about scheduling rules.
The `requiredDuringSchedulingIgnoredDuringExecution` configuration is shown in the table below:
The `preferredDuringSchedulingIgnoredDuringExecution` configuration is shown in the table below:
| Weight | Expression |
| ------ | ------------------------------------------------ |
content/rancher/v2.x/en/helm-charts/legacy-catalogs/globaldns/_index.md
+2
View File
@@ -38,6 +38,7 @@ By default, only [global administrators]({{<baseurl>}}/rancher/v2.x/en/admin-set
1. From the **Global View**, select **Tools > Global DNS Providers**.
1. To add a provider, choose from the available provider options and configure the Global DNS Provider with necessary credentials and an optional domain.
1. (Optional) Add additional users so they could use the provider when creating Global DNS entries as well as manage the Global DNS provider.
1. (Optional) Pass any custom values in the Additional Options section.
{{% accordion id="route53" label="Route53" %}}
1. Enter a **Name** for the provider.
@@ -96,6 +97,7 @@ The [global administrators]({{<baseurl>}}/rancher/v2.x/en/admin-settings/rbac/gl
- Root Domain
- Access Key & Secret Key
- Members
- Custom values
1. From the **Global View**, select **Tools > Global DNS Providers**.
content/rancher/v2.x/en/installation/_index.md
+16 -2
View File
@@ -15,6 +15,8 @@ In this section,
- **The Rancher server** manages and provisions Kubernetes clusters. You can interact with downstream Kubernetes clusters through the Rancher server's user interface.
- **RKE (Rancher Kubernetes Engine)** is a certified Kubernetes distribution and CLI/library which creates and manages a Kubernetes cluster.
- **K3s (Lightweight Kubernetes)** is also a fully compliant Kubernetes distribution. It is newer than RKE, easier to use, and more lightweight, with a binary size of less than 100 MB. As of Rancher v2.4, Rancher can be installed on a K3s cluster.
- **RKE2** is a fully conformant Kubernetes distribution that focuses on security and compliance within the U.S. Federal Government sector.
- **RancherD** is a new tool for installing Rancher, which is available as of Rancher v2.5.4. It is an experimental feature. RancherD is a single binary that first launches an RKE2 Kubernetes cluster, then installs the Rancher server Helm chart on the cluster.
# Changes to Installation in Rancher v2.5
@@ -28,13 +30,25 @@ The `restrictedAdmin` Helm chart option was added. When this option is set to tr
Rancher can be installed on these main architectures:
### High-availability Kubernetes Install
### High-availability Kubernetes Install with the Helm CLI
We recommend using Helm, a Kubernetes package manager, to install Rancher on multiple nodes on a dedicated Kubernetes cluster. For RKE clusters, three nodes are required to achieve a high-availability cluster. For K3s clusters, only two nodes are required.
### High-availability Kubernetes Install with RancherD
_Available as of v2.5.4_
> This is an experimental feature.
RancherD is a single binary that first launches an RKE2 Kubernetes cluster, then installs the Rancher server Helm chart on the cluster.
In both the RancherD install and the Helm CLI install, Rancher is installed as a Helm chart on a Kubernetes cluster.
Configuration and upgrading are also simplified with RancherD. When you upgrade the RancherD binary, both the Kubernetes cluster and the Rancher Helm chart are upgraded.
### Single-node Kubernetes Install
Another option is to install Rancher with Helm on a Kubernetes cluster, but to only use a single node in the cluster. In this case, the Rancher server doesn't have high availability, which is important for running Rancher in production.
Rancher can be installed on a single-node Kubernetes cluster. In this case, the Rancher server doesn't have high availability, which is important for running Rancher in production.
However, this option is useful if you want to save resources by using a single node in the short term, while preserving a high-availability migration path. In the future, you can add nodes to the cluster to get a high-availability Rancher server.
content/rancher/v2.x/en/installation/resources/chart-options/_index.md → content/rancher/v2.x/en/installation/install-rancher-on-k8s/chart-options/_index.md
+8 -1
View File
@@ -1,12 +1,19 @@
---
title: Helm Chart Options
weight: 2
weight: 1
aliases:
- /rancher/v2.x/en/installation/options/
- /rancher/v2.x/en/installation/options/chart-options/
- /rancher/v2.x/en/installation/options/helm2/helm-rancher/chart-options/
- /rancher/v2.x/en/installation/resources/chart-options
---
This page is a configuration reference for the Rancher Helm chart.
For help choosing a Helm chart version, refer to [this page.]({{<baseurl>}}/rancher/v2.x/en/installation/resources/choosing-version/)
For information on enabling experimental features, refer to [this page.]({{<baseurl>}}/rancher/v2.x/en/installation/resources/feature-flags/)
- [Common Options](#common-options)
- [Advanced Options](#advanced-options)
- [API Audit Log](#api-audit-log)
content/rancher/v2.x/en/installation/upgrades-rollbacks/rollbacks/_index.md → content/rancher/v2.x/en/installation/install-rancher-on-k8s/rollbacks/_index.md
+11 -10
View File
@@ -1,22 +1,24 @@
---
title: Rollbacks
weight: 1010
weight: 3
aliases:
- /rancher/v2.x/en/upgrades/rollbacks
- /rancher/v2.x/en/installation/upgrades-rollbacks/rollbacks
- /rancher/v2.x/en/upgrades/ha-server-rollbacks
- /rancher/v2.x/en/upgrades/rollbacks/ha-server-rollbacks
- /rancher/v2.x/en/installation/upgrades-rollbacks/rollbacks/ha-server-rollbacks
- /rancher/v2.x/en/installation/install-rancher-on-k8s/upgrades-rollbacks/rollbacks
---
This section contains information about how to roll back your Rancher server to a previous version.
To roll back to Rancher v2.5+, use the `rancher-backup` application and restore Rancher from backup according to [this section.]({{<baseurl>}}/rancher/v2.x/en/backups/restoring-rancher/)
If you upgrade Rancher and the upgrade does not complete successfully, you may need to [restore Rancher from backup.](../../backups/restores)
To roll back to Rancher prior to v2.5, follow the procedure detailed here: [Restoring Backups — Kubernetes installs]({{<baseurl>}}/rancher/v2.x/en/backups/restorations/ha-restoration) Restoring a snapshot of the Rancher Server cluster will revert Rancher to the version and state at the time of the snapshot.
Restoring a snapshot of the Rancher Server cluster will revert Rancher to the version and state at the time of the snapshot.
For information on how to roll back Rancher installed with Docker, refer to [this page.]({{<baseurl>}}/rancher/v2.x/en/installation/other-installation-methods/single-node-docker/single-node-rollbacks)
>**Note:** Managed clusters are authoritative for their state. This means restoring the rancher server will not revert workload deployments or changes made on managed clusters after the snapshot was taken.
> Managed clusters are authoritative for their state. This means restoring the rancher server will not revert workload deployments or changes made on managed clusters after the snapshot was taken.
- [Rolling back Rancher installed with Docker]({{<baseurl>}}/rancher/v2.x/en/installation/other-installation-methods/single-node-docker/single-node-rollbacks)
- [Rolling back Rancher installed on a Kubernetes cluster]({{<baseurl>}}/rancher/v2.x/en/upgrades/rollbacks/ha-server-rollbacks/)
### Special Scenarios regarding Rollbacks
### Rolling back to v2.0.0-v2.1.5
If you are rolling back to versions in either of these scenarios, you must follow some extra instructions in order to get your clusters working.
@@ -27,7 +29,6 @@ Because of the changes necessary to address [CVE-2018-20321](https://cve.mitre.o
1. Record the `serviceAccountToken` for each cluster. To do this, save the following script on a machine with `kubectl` access to the Rancher management plane and execute it. You will need to run these commands on the machine where the rancher container is running. Ensure JQ is installed before running the command. The commands will vary depending on how you installed Rancher.
**Rancher Installed with Docker**
```
docker exec <NAME OF RANCHER CONTAINER> kubectl get clusters -o json | jq '[.items[] | select(any(.status.conditions[]; .type == "ServiceAccountMigrated")) | {name: .metadata.name, token: .status.serviceAccountToken}]' > tokens.json
content/rancher/v2.x/en/installation/install-rancher-on-k8s/upgrades/_index.md
+288
View File
@@ -0,0 +1,288 @@
---
title: Upgrades
weight: 2
aliases:
- /rancher/v2.x/en/upgrades/upgrades
- /rancher/v2.x/en/installation/upgrades-rollbacks/upgrades
- /rancher/v2.x/en/upgrades/upgrades/ha-server-upgrade-helm-airgap
- /rancher/v2.x/en/upgrades/air-gap-upgrade/
- /rancher/v2.x/en/upgrades/upgrades/ha
- /rancher/v2.x/en/installation/install-rancher-on-k8s/upgrades/upgrades/ha
- /rancher/v2.x/en/installation/upgrades-rollbacks/upgrades/
- /rancher/v2.x/en/upgrades/upgrades/ha-server-upgrade-helm/
- /rancher/v2.x/en/installation/upgrades-rollbacks/upgrades/ha
- /rancher/v2.x/en/installation/install-rancher-on-k8s/upgrades-rollbacks/upgrades
---
The following instructions will guide you through upgrading a Rancher server that was installed on a Kubernetes cluster with Helm. These steps also apply to air gap installs with Helm.
For the instructions to upgrade Rancher installed on Kubernetes with RancherD, refer to [this page.]({{<baseurl>}}/rancher/v2.x/en/installation/install-rancher-on-linux/upgrades)
For the instructions to upgrade Rancher installed with Docker, refer to [ths page.]({{<baseurl>}}/rancher/v2.x/en/installation/other-installation-methods/single-node-docker/single-node-upgrades)
To upgrade the components in your Kubernetes cluster, or the definition of the [Kubernetes services]({{<baseurl>}}/rke/latest/en/config-options/services/) or [add-ons]({{<baseurl>}}/rke/latest/en/config-options/add-ons/), refer to the [upgrade documentation for RKE]({{<baseurl>}}/rke/latest/en/upgrades/), the Rancher Kubernetes Engine.
If you installed Rancher using the RKE Add-on yaml, follow the directions to [migrate or upgrade]({{<baseurl>}}/rancher/v2.x/en/upgrades/upgrades/migrating-from-rke-add-on).
- [Prerequisites](#prerequisites)
- [Upgrade Outline](#upgrade-outline)
- [Known Upgrade Issues](#known-upgrade-issues)
- [RKE Add-on Installs](#rke-add-on-installs)
# Prerequisites
### Access to kubeconfig
Helm should be run from the same location as your kubeconfig file, or the same location where you run your kubectl commands from.
If you installed Kubernetes with RKE, the config will have been created in the directory you ran `rke up` in.
The kubeconfig can also be manually targeted for the intended cluster with the `--kubeconfig` tag (see: https://helm.sh/docs/helm/helm/)
### Review Known Issues
Review the [known upgrade issues](#known-upgrade-issues) in the Rancher documentation for the most noteworthy issues to consider when upgrading Rancher.
A more complete list of known issues for each Rancher version can be found in the release notes on [GitHub](https://github.com/rancher/rancher/releases) and on the [Rancher forums.](https://forums.rancher.com/c/announcements/12)
Note that upgrades _to_ or _from_ any chart in the [rancher-alpha repository]({{<baseurl>}}/rancher/v2.x/en/installation/resources/chart-options/#helm-chart-repositories/) aren't supported.
### Helm Version
The upgrade instructions assume you are using Helm 3.
For migration of installs started with Helm 2, refer to the official [Helm 2 to 3 migration docs.](https://helm.sh/blog/migrate-from-helm-v2-to-helm-v3/) The [Helm 2 upgrade page here]({{<baseurl>}}/rancher/v2.x/en/installation/upgrades-rollbacks/upgrades/ha/helm2)provides a copy of the older upgrade instructions that used Helm 2, and it is intended to be used if upgrading to Helm 3 is not feasible.
### For air gap installs: Populate private registry
-For [air gap installs only,]({{<baseurl>}}/rancher/v2.x/en/installation/other-installation-methods/air-gap) collect and populate images for the new Rancher server version. Follow the guide to [populate your private registry]({{<baseurl>}}/rancher/v2.x/en/installation/other-installation-methods/air-gap/populate-private-registry/) with the images for the Rancher version that you want to upgrade to.
### For upgrades from a Rancher server with a hidden local cluster
If you are upgrading to Rancher v2.5 from a Rancher server that was started with the Helm chart option `--add-local=false`, you will need to drop that flag when upgrading. Otherwise, the Rancher server will not start. The `restricted-admin` role can be used to continue restricting access to the local cluster. For more information, see [this section.]({{<baseurl>}}/rancher/v2.x/en/admin-settings/rbac/global-permissions/#upgrading-from-rancher-with-a-hidden-local-cluster)
### For upgrades from v2.0-v2.2 with external TLS termination
If you are upgrading Rancher from v2.x to v2.3+, and you are using external TLS termination, you will need to edit the cluster.yml to [enable using forwarded host headers.]({{<baseurl>}}/rancher/v2.x/en/installation/resources/chart-options/#configuring-ingress-for-external-tls-when-using-nginx-v0-25)
### For upgrades with cert-manager older than 0.8.0
[Let's Encrypt will be blocking cert-manager instances older than 0.8.0 starting November 1st 2019.](https://community.letsencrypt.org/t/blocking-old-cert-manager-versions/98753) Upgrade cert-manager to the latest version by following [these instructions.]({{<baseurl>}}/rancher/v2.x/en/installation/options/upgrading-cert-manager)
# Upgrade Outline
Follow the steps to upgrade Rancher server:
- [1. Back up your Kubernetes cluster that is running Rancher server](#1-back-up-your-kubernetes-cluster-that-is-running-rancher-server)
- [2. Update the Helm chart repository](#2-update-the-helm-chart-repository)
- [3. Upgrade Rancher](#3-upgrade-rancher)
- [4. Verify the Upgrade](#4-verify-the-upgrade)
# 1. Back up Your Kubernetes Cluster that is Running Rancher Server
For Rancher v2.5+, use the [backup application]({{<baseurl>}}/rancher/v2.x/en/backups/v2.5/back-up-rancher) to back up Rancher.
For Rancher v2.0-v2.4, [take a one-time snapshot]({{<baseurl>}}/rancher/v2.x/en/backups/backups/ha-backups/#option-b-one-time-snapshots)
of your Kubernetes cluster running Rancher server.
You'll use the backup as a restoration point if something goes wrong during upgrade.
# 2. Update the Helm chart repository
1. Update your local helm repo cache.
```
helm repo update
```
1. Get the repository name that you used to install Rancher.
For information about the repos and their differences, see [Helm Chart Repositories]({{<baseurl>}}/rancher/v2.x/en/installation/resources/chart-options/#helm-chart-repositories).
{{< release-channel >}}
```
helm repo list
NAME URL
stable https://kubernetes-charts.storage.googleapis.com
rancher-<CHART_REPO> https://releases.rancher.com/server-charts/<CHART_REPO>
```
> **Note:** If you want to switch to a different Helm chart repository, please follow the [steps on how to switch repositories]({{<baseurl>}}/rancher/v2.x/en/installation/resources/chart-options/#switching-to-a-different-helm-chart-repository). If you switch repositories, make sure to list the repositories again before continuing onto Step 3 to ensure you have the correct one added.
1. Fetch the latest chart to install Rancher from the Helm chart repository.
This command will pull down the latest charts and save it in the current directory as a `.tgz` file.
```plain
helm fetch rancher-<CHART_REPO>/rancher
```
You can fetch the chart for the specific version you are upgrading to by adding in the `--version=` tag. For example:
```plain
helm fetch rancher-<CHART_REPO>/rancher --version=v2.4.11
```
# 3. Upgrade Rancher
This section describes how to upgrade normal (Internet-connected) or air gap installations of Rancher with Helm.
{{% tabs %}}
{{% tab "Kubernetes Upgrade" %}}
Get the values, which were passed with `--set`, from the current Rancher Helm chart that is installed.
```
helm get values rancher -n cattle-system
hostname: rancher.my.org
```
> **Note:** There will be more values that are listed with this command. This is just an example of one of the values.
If you are also upgrading cert-manager to the latest version from a version older than 0.11.0, follow [Option B: Reinstalling Rancher and cert-manager.](#option-b-reinstalling-rancher-and-cert-manager)
Otherwise, follow [Option A: Upgrading Rancher.](#option-a-upgrading-rancher)
### Option A: Upgrading Rancher
Upgrade Rancher to the latest version with all your settings.
Take all the values from the previous step and append them to the command using `--set key=value`:
```
helm upgrade rancher rancher-<CHART_REPO>/rancher \
--namespace cattle-system \
--set hostname=rancher.my.org
```
> **Note:** The above is an example, there may be more values from the previous step that need to be appended.
Alternatively, it's possible to export the current values to a file and reference that file during upgrade. For example, to only change the Rancher version:
```
helm get values rancher -n cattle-system -o yaml > values.yaml
helm upgrade rancher rancher-<CHART_REPO>/rancher \
--namespace cattle-system \
-f values.yaml \
--version=2.4.5
```
### Option B: Reinstalling Rancher and cert-manager
If you are currently running the cert-manger whose version is older than v0.11, and want to upgrade both Rancher and cert-manager to a newer version, then you need to reinstall both Rancher and cert-manger due to the API change in cert-manger v0.11.
1. Uninstall Rancher
```
helm delete rancher -n cattle-system
```
2. Uninstall and reinstall `cert-manager` according to the instructions on the [Upgrading Cert-Manager]({{<baseurl>}}/rancher/v2.x/en/installation/options/upgrading-cert-manager) page.
3. Reinstall Rancher to the latest version with all your settings. Take all the values from the step 1 and append them to the command using `--set key=value`. Note: There will be many more options from the step 1 that need to be appended.
```
helm install rancher rancher-<CHART_REPO>/rancher \
--namespace cattle-system \
--set hostname=rancher.my.org
```
{{% /tab %}}
{{% tab "Kubernetes Air Gap Upgrade" %}}
Render the Rancher template using the same chosen options that were used when installing Rancher. Use the reference table below to replace each placeholder. Rancher needs to be configured to use the private registry in order to provision any Rancher launched Kubernetes clusters or Rancher tools.
Based on the choice you made during installation, complete one of the procedures below.
Placeholder | Description
------------|-------------
`<VERSION>` | The version number of the output tarball.
`<RANCHER.YOURDOMAIN.COM>` | The DNS name you pointed at your load balancer.
`<REGISTRY.YOURDOMAIN.COM:PORT>` | The DNS name for your private registry.
`<CERTMANAGER_VERSION>` | Cert-manager version running on k8s cluster.
### Option A: Default Self-signed Certificate
```plain
helm template ./rancher-<VERSION>.tgz --output-dir . \
--name rancher \
--namespace cattle-system \
--set hostname=<RANCHER.YOURDOMAIN.COM> \
--set certmanager.version=<CERTMANAGER_VERSION> \
--set rancherImage=<REGISTRY.YOURDOMAIN.COM:PORT>/rancher/rancher \
--set systemDefaultRegistry=<REGISTRY.YOURDOMAIN.COM:PORT> \ # Available as of v2.2.0, set a default private registry to be used in Rancher
--set useBundledSystemChart=true # Available as of v2.3.0, use the packaged Rancher system charts
```
### Option B: Certificates from Files using Kubernetes Secrets
```plain
helm template ./rancher-<VERSION>.tgz --output-dir . \
--name rancher \
--namespace cattle-system \
--set hostname=<RANCHER.YOURDOMAIN.COM> \
--set rancherImage=<REGISTRY.YOURDOMAIN.COM:PORT>/rancher/rancher \
--set ingress.tls.source=secret \
--set systemDefaultRegistry=<REGISTRY.YOURDOMAIN.COM:PORT> \ # Available as of v2.2.0, set a default private registry to be used in Rancher
--set useBundledSystemChart=true # Available as of v2.3.0, use the packaged Rancher system charts
```
If you are using a Private CA signed cert, add `--set privateCA=true` following `--set ingress.tls.source=secret`:
```plain
helm template ./rancher-<VERSION>.tgz --output-dir . \
--name rancher \
--namespace cattle-system \
--set hostname=<RANCHER.YOURDOMAIN.COM> \
--set rancherImage=<REGISTRY.YOURDOMAIN.COM:PORT>/rancher/rancher \
--set ingress.tls.source=secret \
--set privateCA=true \
--set systemDefaultRegistry=<REGISTRY.YOURDOMAIN.COM:PORT> \ # Available as of v2.2.0, set a default private registry to be used in Rancher
--set useBundledSystemChart=true # Available as of v2.3.0, use the packaged Rancher system charts
```
### Apply the Rendered Templates
Copy the rendered manifest directories to a system with access to the Rancher server cluster and apply the rendered templates.
Use `kubectl` to apply the rendered manifests.
```plain
kubectl -n cattle-system apply -R -f ./rancher
```
{{% /tab %}}
{{% /tabs %}}
# 4. Verify the Upgrade
Log into Rancher to confirm that the upgrade succeeded.
>**Having network issues following upgrade?**
>
> See [Restoring Cluster Networking]({{<baseurl>}}/rancher/v2.x/en/installation/install-rancher-on-k8s/upgrades/namespace-migration/#restoring-cluster-networking).
# Known Upgrade Issues
The following table lists some of the most noteworthy issues to be considered when upgrading Rancher. A more complete list of known issues for each Rancher version can be found in the release notes on [GitHub](https://github.com/rancher/rancher/releases) and on the [Rancher forums.](https://forums.rancher.com/c/announcements/12)
Upgrade Scenario | Issue
---|---
Upgrading to v2.4.6 or v2.4.7 | These Rancher versions had an issue where the `kms:ListKeys` permission was required to create, edit, or clone Amazon EC2 node templates. This requirement was removed in v2.4.8.
Upgrading to v2.3.0+ | Any user provisioned cluster will be automatically updated upon any edit as tolerations were added to the images used for Kubernetes provisioning.
Upgrading to v2.2.0-v2.2.x | Rancher introduced the [system charts](https://github.com/rancher/system-charts) repository which contains all the catalog items required for features such as monitoring, logging, alerting and global DNS. To be able to use these features in an air gap install, you will need to mirror the `system-charts` repository locally and configure Rancher to use that repository. Please follow the instructions to [configure Rancher system charts]({{<baseurl>}}/rancher/v2.x/en/installation/resources/local-system-charts/#setting-up-system-charts-for-rancher-prior-to-v2-3-0).
Upgrading from v2.0.13 or earlier | If your cluster's certificates have expired, you will need to perform [additional steps]({{<baseurl>}}/rancher/v2.x/en/cluster-admin/certificate-rotation/#rotating-expired-certificates-after-upgrading-older-rancher-versions) to rotate the certificates.
Upgrading from v2.0.7 or earlier | Rancher introduced the `system` project, which is a project that's automatically created to store important namespaces that Kubernetes needs to operate. During upgrade to v2.0.7+, Rancher expects these namespaces to be unassigned from all projects. Before beginning upgrade, check your system namespaces to make sure that they're unassigned to [prevent cluster networking issues]({{<baseurl>}}/rancher/v2.x/en/upgrades/upgrades/namespace-migration/#preventing-cluster-networking-issues).
# RKE Add-on Installs
**Important: RKE add-on install is only supported up to Rancher v2.0.8**
Please use the Rancher helm chart to install Rancher on a Kubernetes cluster. For details, see the [Kubernetes Install - Installation Outline]({{<baseurl>}}/rancher/v2.x/en/installation/install-rancher-on-k8s/#installation-outline).
If you are currently using the RKE add-on install method, see [Migrating from a RKE add-on install]({{<baseurl>}}/rancher/v2.x/en/upgrades/upgrades/migrating-from-rke-add-on/) for details on how to move to using the helm chart.
content/rancher/v2.x/en/installation/upgrades-rollbacks/upgrades/ha/helm2/_index.md → content/rancher/v2.x/en/installation/install-rancher-on-k8s/upgrades/helm2/_index.md
+3
View File
@@ -4,6 +4,9 @@ weight: 1050
aliases:
- /rancher/v2.x/en/upgrades/upgrades/ha/helm2
- /rancher/v2.x/en/upgrades/helm2
- /rancher/v2.x/en/installation/upgrades-rollbacks/upgrades/ha/helm2
- /rancher/v2.x/en/installation/install-rancher-on-k8s/upgrades-rollbacks/upgrades/ha/helm2
- /rancher/v2.x/en/installation/install-rancher-on-k8s/upgrades-rollbacks/upgrades/helm2
---
> Helm 3 has been released. If you are using Helm 2, we recommend [migrating to Helm 3](https://helm.sh/blog/migrate-from-helm-v2-to-helm-v3/) because it is simpler to use and more secure than Helm 2.
content/rancher/v2.x/en/installation/upgrades-rollbacks/upgrades/migrating-from-rke-add-on/_index.md → content/rancher/v2.x/en/installation/install-rancher-on-k8s/upgrades/migrating-from-rke-add-on/_index.md
+2
View File
@@ -5,6 +5,8 @@ aliases:
- /rancher/v2.x/en/upgrades/ha-server-upgrade/
- /rancher/v2.x/en/upgrades/upgrades/ha-server-upgrade/
- /rancher/v2.x/en/upgrades/upgrades/migrating-from-rke-add-on
- /rancher/v2.x/en/installation/upgrades-rollbacks/upgrades/migrating-from-rke-add-on
- /rancher/v2.x/en/installation/install-rancher-on-k8s/upgrades-rollbacks/upgrades/migrating-from-rke-add-on
---
> **Important: RKE add-on install is only supported up to Rancher v2.0.8**
content/rancher/v2.x/en/installation/upgrades-rollbacks/upgrades/namespace-migration/_index.md → content/rancher/v2.x/en/installation/install-rancher-on-k8s/upgrades/namespace-migration/_index.md
+2
View File
@@ -3,6 +3,8 @@ title: Upgrading to v2.0.7+ — Namespace Migration
weight: 1040
aliases:
- /rancher/v2.x/en/upgrades/upgrades/namespace-migration
- /rancher/v2.x/en/installation/upgrades-rollbacks/upgrades/namespace-migration
- /rancher/v2.x/en/installation/install-rancher-on-k8s/upgrades-rollbacks/upgrades/namespace-migration
---
>This section applies only to Rancher upgrades from v2.0.6 or earlier to v2.0.7 or later. Upgrades from v2.0.7 to later version are unaffected.
content/rancher/v2.x/en/installation/install-rancher-on-linux/_index.md
+239
View File
@@ -0,0 +1,239 @@
---
title: Install Rancher on a Linux OS
weight: 2
---
_Available as of Rancher v2.5.4_
> This is an experimental feature.
We are excited to introduce a new, simpler way to install Rancher called RancherD.
RancherD is a single binary that first launches an RKE2 Kubernetes cluster, then installs the Rancher server Helm chart on the cluster.
- [About RancherD Installs](#about-rancherd-installs)
- [Prerequisites](#prerequisites)
- [Part I: Installing Rancher](#part-i-installing-rancher)
- [Part II: High Availability](#part-ii-high-availability)
- [Upgrades](#upgrades)
- [Configuration](#configuration)
- [Uninstall](#uninstall)
- [RKE2 Documentation](#rke2-documentation)
# About RancherD Installs
When RancherD is launched on a host, it first installs an RKE2 Kubernetes cluster, then deploys Rancher on the cluster as a Kubernetes daemonset.
In both the RancherD install and the Helm CLI install, Rancher is installed as a Helm chart on a Kubernetes cluster.
Configuration and upgrading are also simplified with RancherD. When you upgrade the RancherD binary, both the Kubernetes cluster and the Rancher Helm chart are upgraded.
In Part I of these instructions, you'll learn how to launch RancherD on a single node. The result of following the steps in Part I is a single-node [RKE2](https://docs.rke2.io/) Kubernetes cluster with the Rancher server installed. This cluster can easily become high availability later. If Rancher only needs to manage the local Kubernetes cluster, the installation is complete.
Part II explains how to convert the single-node Rancher installation into a high-availability installation. If the Rancher server will manage downstream Kubernetes clusters, it is important to follow these steps. A discussion of recommended architecture for highly available Rancher deployments can be found in our [Best Practices Guide.]({{<baseurl>}}/rancher/v2.x/en/best-practices/v2.5/rancher-server)
# Prerequisites
### Node Requirements
RancherD must be launched on a Linux OS. At this time, only OSes that leverage systemd are supported.
The Linux node needs to fulfill the [installation requirements]({{<baseurl>}}/rancher/v2.x/en/installation/requirements) for hardware and networking. Docker is not required for RancherD installs.
To install RancherD on SELinux Enforcing CentOS 8 nodes or RHEL 8 nodes, some [additional steps]({{<baseurl>}}/rancher/v2.x/en/installation/requirements/#rancherd-on-selinux-enforcing-centos-8-or-rhel-8-nodes) are required.
### Root Access
Before running the installation commands, you will need to log in as root:
```
sudo -s
```
### Fixed Registration Address
A fixed registration address is recommended for single-node installs and required for high-availability installs with RancherD.
The fixed registration address is an endpoint that is used for two purposes:
- To access the Kubernetes API. So you can, for example, modify your [kubeconfig](https://kubernetes.io/docs/concepts/configuration/organize-cluster-access-kubeconfig/) file to point to it instead of a specific node.
- To add new nodes to the Kubernetes cluster. To add nodes to the cluster later, you will run a command on the node that will specify the fixed registration address of the cluster.
If you are installing Rancher on a single node, the fixed registration address makes it possible to add more nodes to the cluster so that you can convert the single-node install to a high-availability install without causing downtime to the cluster. If you don't set up this address when installing the single-node Kubernetes cluster, you would need to re-run the installation script with a fixed registration address in order to add new nodes to the cluster.
The fixed registration can be the IP or hostname of any of the server nodes, but in many cases those may change over time as nodes are created and destroyed. Therefore, you should have a stable endpoint in front of the server nodes.
This endpoint can be set up using any number of approaches, such as:
* A layer 4 (TCP) load balancer
* Round-robin DNS
* Virtual or elastic IP addresses
The following should be taken into consideration when configuring the load balancer or other endpoint:
- The RancherD server process listens on port 9345 for new nodes to register.
- The Kubernetes API is served on port 6443, as normal.
- In RancherD installs, the Rancher UI is served on port 8443 by default. (This is different from Helm chart installs, where port 443 is used by default.)
# Part I: Installing Rancher
### 1. Set up Configurations
To avoid certificate errors with the fixed registration address, you should launch the server with the `tls-san` parameter set. This parameter should refer to your fixed registration address.
This option adds an additional hostname or IP as a Subject Alternative Name in the server's TLS cert, and it can be specified as a list if you would like to access the Kubernetes cluster via both the IP and the hostname.
Create the RancherD config file at `/etc/rancher/rke2/config.yaml`:
```yaml
token: my-shared-secret
tls-san:
- https://my-fixed-registration-address.com
- another-kubernetes-domain.com
```
The first server node establishes the secret token that other nodes would register with if they are added to the cluster.
If you do not specify a pre-shared secret, RancherD will generate one and place it at `/var/lib/rancher/rke2/server/node-token`.
To specify your own pre-shared secret as the token, set the `token` argument on startup.
Installing Rancher this way will use Rancher-generated certificates. To use your own self-signed or trusted certificates, refer to the [configuration guide.]({{<baseurl>}}/rancher/v2.x/en/installation/install-rancher-on-linux/rancherd-configuration/#certificates-for-the-rancher-server)
For information on customizing the RancherD Helm chart values.yaml, refer to [this section.]({{<baseurl>}}/rancher/v2.x/en/installation/install-rancher-on-linux/rancherd-configuration/#customizing-the-rancherd-helm-chart)
### 2. Launch the first server node
Run the RancherD installer:
```
curl -sfL https://get.rancher.io | sh -
```
The RancherD version can be specified using the `INSTALL_RANCHERD_VERSION` environment variable:
```
curl -sfL https://get.rancher.io | INSTALL_RANCHERD_VERSION=v2.5.4-rc6 sh -
```
Once installed, the `rancherd` binary will be on your PATH. You can check out its help text like this:
```
rancherd --help
NAME:
rancherd - Rancher Kubernetes Engine 2
...
```
Next, launch RancherD:
```
systemctl enable rancherd-server.service
systemctl start rancherd-server.service
```
When RancherD launches, it installs an RKE2 Kubernetes cluster. Use the following command to see the logs of the Kubernetes cluster as it comes up:
```
journalctl -eu rancherd-server -f
```
### 3. Set up the kubeconfig file with kubectl
Once the Kubernetes cluster is up, set up RancherD’s kubeconfig file and `kubectl`:
```
export KUBECONFIG=/etc/rancher/rke2/rke2.yaml PATH=$PATH:/var/lib/rancher/rke2/bin
```
### 4. Verify that Rancher is installed on the Kubernetes cluster
Now, you can start issuing `kubectl` commands. Use the following commands to verify that Rancher is deployed as a daemonset on the cluster:
```
kubectl get daemonset rancher -n cattle-system
kubectl get pod -n cattle-system
```
If you watch the pods, you will see the following pods installed:
- `helm-operation` pods in the `cattle-system` namespace
- a `rancher` pod and `rancher-webhook` pod in the `cattle-system` namespace
- a `fleet-agent`, `fleet-controller`, and `gitjob` pod in the `fleet-system` namespace
- a `rancher-operator` pod in the `rancher-operator-system` namespace
### 5. Set the initial Rancher password
Once the `rancher` pod is up and running, run the following:
```
rancherd reset-admin
```
This will give you the URL, username and password needed to log into Rancher. Follow that URL, plug in the credentials, and you’re up and running with Rancher!
If Rancher will only manage the local Kubernetes cluster, the installation is complete.
# Part II: High Availability
If you plan to use the Rancher server to manage downstream Kubernetes clusters, Rancher needs to be highly available. In these steps, you will add more nodes to achieve a high-availability cluster. Since Rancher is running as a daemonset, it will automatically launch on the nodes you add.
An odd number of nodes is required because the etcd cluster, which contains the cluster data, needs a majority of live nodes to avoid losing quorum. A loss of quorum could require the cluster to be restored from backup. Therefore, we recommend using three nodes.
When following these steps, you should still be logged in as root.
### 1. Configure the fixed registration address on a new node
Additional server nodes are launched much like the first, except that you must specify the `server` and `token` parameters so that they can successfully connect to the initial server node.
Here is an example of what the RancherD config file would look like for additional server nodes. By default, this config file is expected to be located at `/etc/rancher/rke2/config.yaml`.
```yaml
server: https://my-fixed-registration-address.com:9345
token: my-shared-secret
tls-san:
- my-fixed-registration-address.com
- another-kubernetes-domain.com
```
### 2. Launch an additional server node
Run the installer on the new node:
```
curl -sfL https://get.rancher.io | sh -
```
This will download RancherD and install it as a systemd unit on your host.
Next, launch RancherD:
```
systemctl enable rancherd-server.service
systemctl start rancherd-server.service
```
### 3. Repeat
Repeat steps one and two for another Linux node, bringing the number of nodes in the cluster to three.
**Result:** Rancher is highly available and the installation is complete.
# Upgrades
For information on upgrades and rollbacks, refer to [this page.](./upgrades)
# Configuration
For information on how to configure certificates, node taints, Rancher Helm chart options, or RancherD CLI options, refer to the [configuration reference.](./rancherd-configuration)
# Uninstall
To uninstall RancherD from your system, run the command below. This will shut down the process, remove the RancherD binary, and clean up files used by RancherD.
```
rancherd-uninstall.sh
```
# RKE2 Documentation
For more information on RKE2, the Kubernetes distribution used to provision the underlying cluster, refer to the documentation [here.](https://docs.rke2.io/)
content/rancher/v2.x/en/installation/install-rancher-on-linux/rancherd-configuration/_index.md
+316
View File
@@ -0,0 +1,316 @@
---
title: RancherD Configuration Reference
weight: 1
---
In RancherD, a server node is defined as a machine (bare-metal or virtual) running the `rancherd server` command. The server runs the Kubernetes API as well as Kubernetes workloads.
An agent node is defined as a machine running the `rancherd agent` command. They don't run the Kubernetes API. To add nodes designated to run your apps and services, join agent nodes to your cluster.
In the RancherD installation instructions, we recommend running three server nodes in the Rancher server cluster. Agent nodes are not required.
- [Certificates for the Rancher Server](#certificates-for-the-rancher-server)
- [Node Taints](#node-taints)
- [Customizing the RancherD Helm Chart](#customizing-the-rancherd-helm-chart)
- [RancherD Server CLI Options](#rancherd-server-cli-options)
- [RancherD Agent CLI Options](#rancherd-agent-cli-options)
# Certificates for the Rancher Server
Rancherd does not use cert-manger to provision certs. Instead RancherD allows you to bring your own self-signed or trusted certs by storing the .pem files in `/etc/rancher/ssl/`. When doing this you should also set the `publicCA` parameter to `true` in your HelmChartConfig. For more information on the HelmChartConfig, refer to the section about [customizing the RancherD Helm chart.](#customizing-the-rancherd-helm-chart)
Private key: `/etc/rancher/ssl/key.pem`
Certificate: `/etc/rancher/ssl/cert.pem`
CA Certificate(self-signed): `/etc/rancher/ssl/cacerts.pem`
Additional CA Certificate: `/etc/ssl/certs/ca-additional.pem`
# Node Taints
By default, server nodes will be schedulable and thus your workloads can get launched on them. If you wish to have a dedicated control plane where no user workloads will run, you can use taints. The node-taint parameter will allow you to configure nodes with taints. Here is an example of adding a node taint to the `config.yaml`:
```
node-taint:
- "CriticalAddonsOnly=true:NoExecute"
```
# Customizing the RancherD Helm Chart
Rancher is launched as a [Helm](https://helm.sh/) chart using the cluster’s [Helm integration.](https://docs.rke2.io/helm/) This means that you can easily customize the application through a manifest file describing your custom parameters.
The RancherD chart provisions Rancher in a daemonset. It exposes hostport `8080/8443` down to the container port (`80/443`), and uses hostpath to mount certs if needed.
RancherD uses `helm-controller` to bootstrap the RancherD chart. To provide a customized `values.yaml` file, the configuration options must be passed in through the `helm-controller` custom resource definition.
Here is an example of the manifest:
```yaml
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
name: rancher
namespace: kube-system
spec:
valuesContent: |
publicCA: true
```
Put this manifest on your host in `/var/lib/rancher/rke2/server/manifests` before running RancherD.
### Common Options
| Parameter | Default Value | Description |
| ------------------------------ | ----------------------------------------------------- | -------------------------------------------- |
| `addLocal` | "auto" | ***string*** - Have Rancher detect and import the “local” Rancher server cluster [Import "local Cluster"](https://rancher.com/docs/rancher/v2.x/en/installation/options/chart-options/#import-local-cluster) |
| `auditLog.destination` | "sidecar" | ***string*** - Stream to sidecar container console or hostPath volume - *"sidecar, hostPath"* |
| `auditLog.hostPath` | "/var/log/rancher/audit" | ***string*** - log file destination on host (only applies when **auditLog.destination** is set to **hostPath**) |
| `auditLog.level` | 0 | ***int*** - set the [API Audit Log level](https://rancher.com/docs/rancher/v2.x/en/installation/api-auditing). 0 is off. [0-3] |
| `auditLog.maxAge` | 1 | ***int*** - maximum number of days to retain old audit log files (only applies when **auditLog.destination** is set to **hostPath**) |
| `auditLog.maxBackups` | 1 | int - maximum number of audit log files to retain (only applies when **auditLog.destination** is set to **hostPath**) |
| `auditLog.maxSize` | 100 | ***int*** - maximum size in megabytes of the audit log file before it gets rotated (only applies when **auditLog.destination** is set to **hostPath**) |
| `debug` | false | ***bool*** - set debug flag on rancher server |
| `extraEnv` | [] | ***list*** - set additional environment variables for Rancher Note: *Available as of v2.2.0* |
| `imagePullSecrets` | [] | ***list*** - list of names of Secret resource containing private registry credentials |
| `proxy` | " " | ***string** - HTTP[S] proxy server for Rancher |
| `noProxy` | "127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16" | ***string*** - comma separated list of hostnames or ip address not to use the proxy |
| `resources` | {} | ***map*** - rancher pod resource requests & limits |
| `rancherImage` | "rancher/rancher" | ***string*** - rancher image source |
| `rancherImageTag` | same as chart version | ***string*** - rancher/rancher image tag |
| `rancherImagePullPolicy` | "IfNotPresent" | ***string*** - Override imagePullPolicy for rancher server images - *"Always", "Never", "IfNotPresent"* |
| `systemDefaultRegistry` | "" | ***string*** - private registry to be used for all system Docker images, e.g., [http://registry.example.com/] *Available as of v2.3.0* |
| `useBundledSystemChart` | false | ***bool*** - select to use the system-charts packaged with Rancher server. This option is used for air gapped installations. *Available as of v2.3.0* |
| `publicCA` | false | ***bool*** - Set to true if your cert is signed by a public CA |
# RancherD Server CLI Options
The command to run the Rancher management server is:
```
rancherd server [OPTIONS]
```
It can be run with the following options:
### Config
| Option | Description |
|--------|-------------|
| `--config FILE, -c FILE` | Load configuration from FILE (default: "/etc/rancher/rke2/config.yaml") |
### Logging
| Option | Description |
|--------|-------------|
| `--debug` | Turn on debug logs |
### Listener
| Option | Description |
|--------|-------------|
| `--bind-address value` | RancherD bind address (default: 0.0.0.0) |
| `--advertise-address value` | IP address that apiserver uses to advertise to members of the cluster (default: node-external-ip/node-ip) |
| `--tls-san value` | Add additional hostname or IP as a Subject Alternative Name in the TLS cert |
### Data
| Option | Description |
|--------|-------------|
| `--data-dir value, -d value` | Folder to hold state (default: "/var/lib/rancher/rancherd") |
### Networking
| Option | Description |
|--------|-------------|
| `--cluster-cidr value` | Network CIDR to use for pod IPs (default: "10.42.0.0/16") |
| `--service-cidr value` | Network CIDR to use for services IPs (default: "10.43.0.0/16") |
| `--cluster-dns value` | Cluster IP for coredns service. Should be in your service-cidr range (default: 10.43.0.10) |
| `--cluster-domain value` | Cluster Domain (default: "cluster.local") |
### Cluster
| Option | Description |
|--------|-------------|
| `--token value, -t value` | Shared secret used to join a server or agent to a cluster |
| `--token-file value` | File containing the cluster-secret/token |
### Client
| Option | Description |
|--------|-------------|
| `--write-kubeconfig value, -o value` | Write kubeconfig for admin client to this file |
| `--write-kubeconfig-mode value` | Write kubeconfig with this mode |
### Flags
| Option | Description |
|--------|-------------|
| `--kube-apiserver-arg value` | Customized flag for kube-apiserver process |
| `--kube-scheduler-arg value` | Customized flag for kube-scheduler process |
| `--kube-controller-manager-arg value` | Customized flag for kube-controller-manager process |
### Database
| Option | Description |
|--------|-------------|
| `--etcd-disable-snapshots` | Disable automatic etcd snapshots |
| `--etcd-snapshot-schedule-cron value` | Snapshot interval time in cron spec. eg. every 5 hours '* */5 * * *' (default: "0 */12 * * *") |
| `--etcd-snapshot-retention value` | Number of snapshots to retain (default: 5) |
| `--etcd-snapshot-dir value` | Directory to save db snapshots. (Default location: ${data-dir}/db/snapshots) |
| `--cluster-reset-restore-path value` | Path to snapshot file to be restored |
### System Images Registry
| Option | Description |
|--------|-------------|
| `--system-default-registry value` | Private registry to be used for all system Docker images |
### Components
| Option | Description |
|--------|-------------|
| `--disable value` | Do not deploy packaged components and delete any deployed components (valid items: rancherd-canal, rancherd-coredns, rancherd-ingress, rancherd-kube-proxy, rancherd-metrics-server) |
### Cloud Provider
| Option | Description |
|--------|-------------|
| `--cloud-provider-name value` | Cloud provider name |
| `--cloud-provider-config value` | Cloud provider configuration file path |
### Security
| Option | Description |
|--------|-------------|
| `--profile value` | Validate system configuration against the selected benchmark (valid items: cis-1.5) |
### Agent Node
| Option | Description |
|--------|-------------|
| `--node-name value` | Node name |
| `--node-label value` | Registering and starting kubelet with set of labels |
| `--node-taint value` | Registering kubelet with set of taints |
| `--protect-kernel-defaults` | Kernel tuning behavior. If set, error if kernel tunables are different than kubelet defaults. |
| `--selinux` | Enable SELinux in containerd |
### Agent Runtime
| Option | Description |
|--------|-------------|
| `--container-runtime-endpoint value` | Disable embedded containerd and use alternative CRI implementation |
| `--snapshotter value` | Override default containerd snapshotter (default: "overlayfs") |
| `--private-registry value` | Private registry configuration file (default: "/etc/rancher/rke2/registries.yaml") |
### Agent Networking
| Option | Description |
|--------|-------------|
| `--node-ip value, -i value` | IP address to advertise for node |
| `--resolv-conf value` | Kubelet resolv.conf file |
### Agent Flags
| Option | Description |
|--------|-------------|
| `--kubelet-arg value` | Customized flag for kubelet process |
| `--kube-proxy-arg value` | Customized flag for kube-proxy process |
### Experimental
| Option | Description |
|--------|-------------|
| `--agent-token value` | Shared secret used to join agents to the cluster, but not servers |
| `--agent-token-file value` | File containing the agent secret |
| `--server value, -s value` | Server to connect to, used to join a cluster |
| `--cluster-reset` | Forget all peers and become sole member of a new cluster |
| `--secrets-encryption` | Enable Secret encryption at rest |
# RancherD Agent CLI Options
The following command is used to run the RancherD agent:
```
rancherd agent [OPTIONS]
```
The following options are available.
### Config
| Option | Description |
|--------|-------------|
| `--config FILE, -c FILE` | Load configuration from FILE (default: "/etc/rancher/rke2/config.yaml") |
### Data
| Option | Description |
|--------|-------------|
| `--data-dir value, -d value` | Folder to hold state (default: "/var/lib/rancher/rancherd") |
### Logging
| Option | Description |
|--------|-------------|
| `--debug` | Turn on debug logs |
### Cluster
| Option | Description |
|--------|-------------|
| `--token value, -t value` | Token to use for authentication |
| `--token-file value` | Token file to use for authentication |
| `--server value, -s value` | Server to connect to |
### Agent Node
| Option | Description |
|--------|-------------|
| `--node-name value` | Node name |
| `--node-label value` | Registering and starting kubelet with set of labels |
| `--node-taint value` | Registering kubelet with set of taints |
| `--selinux` | Enable SELinux in containerd |
| `--protect-kernel-defaults` | Kernel tuning behavior. If set, error if kernel tunables are different than kubelet defaults. |
### Agent Runtime
| Option | Description |
|--------|-------------|
| `--container-runtime-endpoint value` | Disable embedded containerd and use alternative CRI implementation |
| `--snapshotter value` | Override default containerd snapshotter (default: "overlayfs") |
| `--private-registry value` | Private registry configuration file (default: "/etc/rancher/rke2/registries.yaml") |
### Agent Networking
| Option | Description |
|--------|-------------|
| `--node-ip value, -i value` | IP address to advertise for node |
| `--resolv-conf value` | Kubelet resolv.conf file |
### Agent Flags
| Option | Description |
|--------|-------------|
| `--kubelet-arg value` | Customized flag for kubelet process |
| `--kube-proxy-arg value` | Customized flag for kube-proxy process |
### System Images Registry
| Option | Description |
|--------|-------------|
| `--system-default-registry value` | Private registry to be used for all system Docker images |
### Cloud Provider
| Option | Description |
|--------|-------------|
| `--cloud-provider-name value` | Cloud provider name |
| `--cloud-provider-config value` | Cloud provider configuration file path |
### Security
| Option | Description |
|--------|-------------|
| `--profile value` | Validate system configuration against the selected benchmark (valid items: cis-1.5) |
content/rancher/v2.x/en/installation/install-rancher-on-linux/rollbacks/_index.md
+6
View File
@@ -0,0 +1,6 @@
---
title: Rollbacks
weight: 3
---
To roll back Rancher to a previous version, re-run the installation script with the previous version specified in the `INSTALL_RANCHERD_VERSION` environment variable.
content/rancher/v2.x/en/installation/install-rancher-on-linux/upgrades/_index.md
+68
View File
@@ -0,0 +1,68 @@
---
title: Upgrades
weight: 2
---
When RancherD is upgraded, the Rancher Helm controller and the Fleet pods are upgraded.
During a RancherD upgrade, there is very little downtime, but it is possible that RKE2 may be down for a minute, during which you could lose access to Rancher.
When Rancher is installed with RancherD, the underlying Kubernetes cluster can't be upgraded from the Rancher UI. It needs to be upgraded using the RancherD CLI.
### Upgrading the Rancher Helm Chart without Upgrading the Underlying Cluster
To upgrade Rancher without upgrading the underlying Kubernetes cluster, follow these steps.
> Before upgrading, we recommend that you should:
>
> - Create a backup of the Rancher server using the [backup application.]({{<baseurl>}}/rancher/v2.x/en/backups/v2.5/back-up-rancher/)
> - Review the known issues for the Rancher version you are upgrading to. The known issues are listed in the release notes on [GitHub](https://github.com/rancher/rancher/releases) and on the [Rancher forums.](https://forums.rancher.com/c/announcements/12)
1. Uninstall the chart with Helm:
```
helm uninstall rancher
```
2. Reinstall the Rancher chart with Helm. To install a specific Rancher version, use the `--version` flag. For example:
```
helm install rancher rancher-latest/rancher \
--namespace cattle-system \
--set hostname=rancher.my.org \
--version 2.5.1
```
**Result:** Rancher is upgraded to the new version.
If necessary, restore Rancher from backup by following [these steps.]({{<baseurl>}}/rancher/v2.x/en/backups/v2.5/restoring-rancher/)
### Upgrading Both Rancher and the Underlying Cluster
Upgrade both RancherD and the underlying Kubernetes cluster by re-running the RancherD installation script.
> Before upgrading, we recommend that you should:
>
> - Create a backup of the Rancher server using the [backup application.]({{<baseurl>}}/rancher/v2.x/en/backups/v2.5/back-up-rancher/)
> - Review the known issues for the Rancher version you are upgrading to. The known issues are listed in the release notes on [GitHub](https://github.com/rancher/rancher/releases) and on the [Rancher forums.](https://forums.rancher.com/c/announcements/12)
```
sudo curl -sfL https://get.rancher.io | sudo sh -
```
To specify a specific version to upgrade to, use `INSTALL_RANCHERD_VERSION` environment variable:
```
curl -sfL https://get.rancher.io | INSTALL_RANCHERD_VERSION=v2.5.1 sh -
```
Then launch the server:
```
systemctl enable rancherd-server
systemctl start rancherd-server
```
The upgrade can also be performed by manually installing the binary of the desired version.
content/rancher/v2.x/en/installation/other-installation-methods/air-gap/_index.md
+2 -2
View File
@@ -1,5 +1,5 @@
---
title: Installing Rancher in an Air Gapped Environment
title: Air Gapped Helm CLI Install
weight: 1
aliases:
- /rancher/v2.x/en/installation/air-gap-installation/
@@ -7,7 +7,7 @@ aliases:
- /rancher/v2.x/en/installation/air-gap-single-node/
---
This section is about installations of Rancher server in an air gapped environment. An air gapped environment could be where Rancher server will be installed offline, behind a firewall, or behind a proxy.
This section is about using the Helm CLI to install the Rancher server in an air gapped environment. An air gapped environment could be where Rancher server will be installed offline, behind a firewall, or behind a proxy.
The installation steps differ depending on whether Rancher is installed on an RKE Kubernetes cluster, a K3s Kubernetes cluster, or a single Docker container.
content/rancher/v2.x/en/installation/requirements/_index.md
+33 -177
View File
@@ -27,14 +27,10 @@ The Rancher UI works best in Firefox or Chrome.
Rancher should work with any modern Linux distribution.
Docker is required for nodes that will run K3s or RKE Kubernetes clusters. It is not required for RancherD installs.
For details on which OS and Docker versions were tested with each Rancher version, refer to the [support maintenance terms.](https://rancher.com/support-maintenance-terms/)
For the container runtime, RKE should work with any modern Docker version, while K3s should work with any modern version of Docker or containerd.
If you are installing Rancher on a K3s cluster with **Raspbian Buster**, follow [these steps]({{<baseurl>}}/k3s/latest/en/advanced/#enabling-legacy-iptables-on-raspbian-buster) to switch to legacy iptables.
If you are installing Rancher on a K3s cluster with Alpine Linux, follow [these steps]({{<baseurl>}}/k3s/latest/en/advanced/#additional-preparation-for-alpine-linux-setup) for additional setup.
All supported operating systems are 64-bit x86.
The `ntp` (Network Time Protocol) package should be installed. This prevents errors with certificate validation that can occur when the time is not synchronized between the client and server.
@@ -43,10 +39,33 @@ Some distributions of Linux may have default firewall rules that block communica
If you plan to run Rancher on ARM64, see [Running on ARM64 (Experimental).]({{<baseurl>}}/rancher/v2.x/en/installation/options/arm64-platform/)
### RKE Specific Requirements
For the container runtime, RKE should work with any modern Docker version.
### K3s Specific Requirements
For the container runtime, K3s should work with any modern version of Docker or containerd.
If you are installing Rancher on a K3s cluster with **Raspbian Buster**, follow [these steps]({{<baseurl>}}/k3s/latest/en/advanced/#enabling-legacy-iptables-on-raspbian-buster) to switch to legacy iptables.
If you are installing Rancher on a K3s cluster with Alpine Linux, follow [these steps]({{<baseurl>}}/k3s/latest/en/advanced/#additional-preparation-for-alpine-linux-setup) for additional setup.
### RancherD Specific Requirements
_The RancherD install is available as of v2.5.4. It is an experimental feature._
At this time, only Linux OSes that leverage systemd are supported.
To install RancherD on SELinux Enforcing CentOS 8 or RHEL 8 nodes, some [additional steps](#rancherd-on-selinux-enforcing-centos-8-or-rhel-8-nodes) are required.
Docker is not required for RancherD installs.
### Installing Docker
Docker can be installed by following the steps in the official [Docker documentation.](https://docs.docker.com/) Rancher also provides [scripts]({{<baseurl>}}/rancher/v2.x/en/installation/requirements/installing-docker) to install Docker with one command.
Docker is required for Helm chart installs, and it can be installed by following the steps in the official [Docker documentation.](https://docs.docker.com/) Rancher also provides [scripts]({{<baseurl>}}/rancher/v2.x/en/installation/requirements/installing-docker) to install Docker with one command.
Docker is not required for RancherD installs.
# Hardware Requirements
This section describes the CPU, memory, and disk requirements for the nodes where the Rancher server is installed.
@@ -129,176 +148,13 @@ Each node used should have a static IP configured, regardless of whether you are
### Port Requirements
This section describes the port requirements for nodes running the `rancher/rancher` container.
To operate properly, Rancher requires a number of ports to be open on Rancher nodes and on downstream Kubernetes cluster nodes. [Port Requirements]({{<baseurl>}}/rancher/v2.x/en/installation/requirements/ports) lists all the necessary ports for Rancher and Downstream Clusters for the different cluster types.
The port requirements are different depending on whether you are installing Rancher on a K3s cluster, on an RKE cluster, or in a single Docker container.
# RancherD on SELinux Enforcing CentOS 8 or RHEL 8 Nodes
{{% tabs %}}
{{% tab "K3s" %}}
### Ports for Communication with Downstream Clusters
Before installing Rancher on SELinux Enforcing CentOS 8 nodes or RHEL 8 nodes, you must install `container-selinux` and `iptables`:
To communicate with downstream clusters, Rancher requires different ports to be open depending on the infrastructure you are using.
For example, if you are deploying Rancher on nodes hosted by an infrastructure provider, port `22` must be open for SSH.
The following diagram depicts the ports that are opened for each [cluster type]({{<baseurl>}}/rancher/v2.x/en/cluster-provisioning).
<figcaption>Port Requirements for the Rancher Management Plane</figcaption>
![Basic Port Requirements]({{<baseurl>}}/img/rancher/port-communications.svg)
The following tables break down the port requirements for inbound and outbound traffic:
<figcaption>Inbound Rules for Rancher Nodes</figcaption>
| Protocol | Port | Source | Description |
| -------- | ---- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------- |
| TCP | 80 | Load balancer/proxy that does external SSL termination | Rancher UI/API when external SSL termination is used |
| TCP | 443 | <ul><li>server nodes</li><li>agent nodes</li><li>hosted/imported Kubernetes</li><li>any source that needs to be able to use the Rancher UI or API</li></ul> | Rancher agent, Rancher UI/API, kubectl |
<figcaption>Outbound Rules for Rancher Nodes</figcaption>
| Protocol | Port | Destination | Description |
| -------- | ---- | -------------------------------------------------------- | --------------------------------------------- |
| TCP | 22 | Any node IP from a node created using Node Driver | SSH provisioning of nodes using Node Driver |
| TCP | 443 | `35.160.43.145/32`, `35.167.242.46/32`, `52.33.59.17/32` | git.rancher.io (catalogs) |
| TCP | 2376 | Any node IP from a node created using Node driver | Docker daemon TLS port used by Docker Machine |
| TCP | 6443 | Hosted/Imported Kubernetes API | Kubernetes API server |
**Note** Rancher nodes may also require additional outbound access for any external [authentication provider]({{<baseurl>}}/rancher/v2.x/en/admin-settings/authentication/) which is configured (LDAP for example).
### Additional Port Requirements for Nodes in a K3s Kubernetes Cluster
You will need to open additional ports to launch the Kubernetes cluster that is required for a high-availability installation of Rancher.
The K3s server needs port 6443 to be accessible by the nodes.
The nodes need to be able to reach other nodes over UDP port 8472 when Flannel VXLAN is used. The node should not listen on any other port. K3s uses reverse tunneling such that the nodes make outbound connections to the server and all kubelet traffic runs through that tunnel. However, if you do not use Flannel and provide your own custom CNI, then port 8472 is not needed by K3s.
If you wish to utilize the metrics server, you will need to open port 10250 on each node.
> **Important:** The VXLAN port on nodes should not be exposed to the world as it opens up your cluster network to be accessed by anyone. Run your nodes behind a firewall/security group that disables access to port 8472.
<figcaption>Inbound Rules for Rancher Server Nodes</figcaption>
| Protocol | Port | Source | Description
|-----|-----|----------------|---|
| TCP | 6443 | K3s server nodes | Kubernetes API
| UDP | 8472 | K3s server and agent nodes | Required only for Flannel VXLAN
| TCP | 10250 | K3s server and agent nodes | kubelet
Typically all outbound traffic is allowed.
{{% /tab %}}
{{% tab "RKE" %}}
### Ports for Communication with Downstream Clusters
To communicate with downstream clusters, Rancher requires different ports to be open depending on the infrastructure you are using.
For example, if you are deploying Rancher on nodes hosted by an infrastructure provider, port `22` must be open for SSH.
The following diagram depicts the ports that are opened for each [cluster type]({{<baseurl>}}/rancher/v2.x/en/cluster-provisioning).
<figcaption>Port Requirements for the Rancher Management Plane</figcaption>
![Basic Port Requirements]({{<baseurl>}}/img/rancher/port-communications.svg)
The following tables break down the port requirements for inbound and outbound traffic:
<figcaption>Inbound Rules for Rancher Nodes</figcaption>
| Protocol | Port | Source | Description |
| -------- | ---- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------- |
| TCP | 80 | Load balancer/proxy that does external SSL termination | Rancher UI/API when external SSL termination is used |
| TCP | 443 | <ul><li>etcd nodes</li><li>controlplane nodes</li><li>worker nodes</li><li>hosted/imported Kubernetes</li><li>any source that needs to be able to use the Rancher UI or API</li></ul> | Rancher agent, Rancher UI/API, kubectl |
<figcaption>Outbound Rules for Rancher Nodes</figcaption>
| Protocol | Port | Destination | Description |
| -------- | ---- | -------------------------------------------------------- | --------------------------------------------- |
| TCP | 22 | Any node IP from a node created using Node Driver | SSH provisioning of nodes using Node Driver |
| TCP | 443 | `35.160.43.145/32`, `35.167.242.46/32`, `52.33.59.17/32` | git.rancher.io (catalogs) |
| TCP | 2376 | Any node IP from a node created using Node driver | Docker daemon TLS port used by Docker Machine |
| TCP | 6443 | Hosted/Imported Kubernetes API | Kubernetes API server |
**Note** Rancher nodes may also require additional outbound access for any external [authentication provider]({{<baseurl>}}/rancher/v2.x/en/admin-settings/authentication/) which is configured (LDAP for example).
### Additional Port Requirements for Nodes in an RKE Kubernetes Cluster
You will need to open additional ports to launch the Kubernetes cluster that is required for a high-availability installation of Rancher.
If you follow the Rancher installation documentation for setting up a Kubernetes cluster using RKE, you will set up a cluster in which all three nodes have all three roles: etcd, controlplane, and worker. In that case, you can refer to this list of requirements for each node with all three roles.
If you installed Rancher on a Kubernetes cluster that doesn't have all three roles on each node, refer to the [port requirements for the Rancher Kubernetes Engine (RKE).]({{<baseurl>}}/rke/latest/en/os/#ports) The RKE docs show a breakdown of the port requirements for each role.
<figcaption>Inbound Rules for Nodes with All Three Roles: etcd, Controlplane, and Worker</figcaption>
Protocol | Port | Source | Description
-----------|------|----------|--------------
TCP | 22 | Linux worker nodes only, and any network that you want to be able to remotely access this node from. | Remote access over SSH
TCP | 80 | Any source that consumes Ingress services | Ingress controller (HTTP)
TCP | 443 | Any source that consumes Ingress services | Ingress controller (HTTPS)
TCP | 2376 | Rancher nodes | Docker daemon TLS port used by Docker Machine (only needed when using Node Driver/Templates)
TCP | 2379 | etcd nodes and controlplane nodes | etcd client requests
TCP | 2380 | etcd nodes and controlplane nodes | etcd peer communication
TCP | 3389 | Windows worker nodes only, and any network that you want to be able to remotely access this node from. | Remote access over RDP
TCP | 6443 | etcd nodes, controlplane nodes, and worker nodes | Kubernetes apiserver
UDP | 8472 | etcd nodes, controlplane nodes, and worker nodes | Canal/Flannel VXLAN overlay networking
TCP | 9099 | the node itself (local traffic, not across nodes) | Canal/Flannel livenessProbe/readinessProbe
TCP | 10250 | controlplane nodes | kubelet
TCP | 10254 | the node itself (local traffic, not across nodes) | Ingress controller livenessProbe/readinessProbe
TCP/UDP | 30000-32767 | Any source that consumes NodePort services | NodePort port range
<figcaption>Outbound Rules for Nodes with All Three Roles: etcd, Controlplane, and Worker</figcaption>
Protocol | Port | Source | Destination | Description
-----------|------|----------|---------------|--------------
TCP | 22 | RKE node | Any node configured in Cluster Configuration File | SSH provisioning of node by RKE
TCP | 443 | Rancher nodes | Rancher agent |
TCP | 2379 | etcd nodes | etcd client requests |
TCP | 2380 | etcd nodes | etcd peer communication |
TCP | 6443 | RKE node | controlplane nodes | Kubernetes API server
TCP | 6443 | controlplane nodes | Kubernetes API server |
UDP | 8472 | etcd nodes, controlplane nodes, and worker nodes | Canal/Flannel VXLAN overlay networking |
TCP | 9099 | the node itself (local traffic, not across nodes) | Canal/Flannel livenessProbe/readinessProbe |
TCP | 10250 | etcd nodes, controlplane nodes, and worker nodes | kubelet |
TCP | 10254 | the node itself (local traffic, not across nodes) | Ingress controller livenessProbe/readinessProbe
{{% /tab %}}
{{% tab "Docker" %}}
### Ports for Communication with Downstream Clusters
For a Docker installation, you only need to open the ports required to enable Rancher to communicate with downstream user clusters.
The port requirements depend on the infrastructure you are using. For example, if you are deploying Rancher on nodes hosted by an infrastructure provider, port `22` must be open for SSH.
The following diagram depicts the ports that are opened for each [cluster type]({{<baseurl>}}/rancher/v2.x/en/cluster-provisioning).
<figcaption>Port Requirements for the Rancher Management Plane</figcaption>
![Basic Port Requirements]({{<baseurl>}}/img/rancher/port-communications.svg)
The following tables break down the port requirements for Rancher nodes, for inbound and outbound traffic:
**Note** Rancher nodes may also require additional outbound access for any external [authentication provider]({{<baseurl>}}/rancher/v2.x/en/admin-settings/authentication/) which is configured (LDAP for example).
<figcaption>Inbound Rules</figcaption>
| Protocol | Port | Source | Description |
| -------- | ---- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------- |
| TCP | 80 | Load balancer/proxy that does external SSL termination | Rancher UI/API when external SSL termination is used |
| TCP | 443 | <ul><li>etcd nodes</li><li>controlplane nodes</li><li>worker nodes</li><li>hosted/imported Kubernetes</li><li>any source that needs to be able to use the Rancher UI or API</li></ul> | Rancher agent, Rancher UI/API, kubectl |
<figcaption>Outbound Rules</figcaption>
| Protocol | Port | Source | Description |
| -------- | ---- | -------------------------------------------------------- | --------------------------------------------- |
| TCP | 22 | Any node IP from a node created using Node Driver | SSH provisioning of nodes using Node Driver |
| TCP | 443 | `35.160.43.145/32`, `35.167.242.46/32`, `52.33.59.17/32` | git.rancher.io (catalogs) |
| TCP | 2376 | Any node IP from a node created using Node driver | Docker daemon TLS port used by Docker Machine |
| TCP | 6443 | Hosted/Imported Kubernetes API | Kubernetes API server |
**Note** Rancher nodes may also require additional outbound access for any external [authentication provider]({{<baseurl>}}/rancher/v2.x/en/admin-settings/authentication/) which is configured (LDAP for example).
{{% /tab %}}
{{% /tabs %}}
```
sudo yum install iptables
sudo yum install container-selinux
```
content/rancher/v2.x/en/installation/requirements/ports/_index.md
+153 -48
View File
@@ -6,14 +6,39 @@ weight: 300
To operate properly, Rancher requires a number of ports to be open on Rancher nodes and on downstream Kubernetes cluster nodes.
## Rancher Nodes
- [Rancher Nodes](#rancher-nodes)
- [Ports for Rancher Server Nodes on K3s](#ports-for-rancher-server-nodes-on-k3s)
- [Ports for Rancher Server Nodes on RKE](#ports-for-rancher-server-nodes-on-rke)
- [Ports for Rancher Server Nodes on RancherD or RKE2](#ports-for-rancher-server-nodes-on-rancherd-or-rke2)
- [Ports for Rancher Server in Docker](#ports-for-rancher-server-in-docker)
- [Downstream Kubernetes Cluster Nodes](#downstream-kubernetes-cluster-nodes)
- [Ports for Rancher Launched Kubernetes Clusters using Node Pools](#ports-for-rancher-launched-kubernetes-clusters-using-node-pools)
- [Ports for Rancher Launched Kubernetes Clusters using Custom Nodes](#ports-for-rancher-launched-kubernetes-clusters-using-custom-nodes)
- [Ports for Hosted Kubernetes Clusters](#ports-for-hosted-kubernetes-clusters)
- [Ports for Registered Clusters](#ports-for-registered-clusters)
- [Other Port Considerations](#other-port-considerations)
- [Commonly Used Ports](#commonly-used-ports)
- [Local Node Traffic](#local-node-traffic)
- [Rancher AWS EC2 Security Group](#rancher-aws-ec2-security-group)
- [Opening SUSE Linux Ports](#opening-suse-linux-ports)
# Rancher Nodes
The following table lists the ports that need to be open to and from nodes that are running the Rancher server.
The port requirements differ based on whether Rancher is installed in a K3s Kubernetes cluster, an RKE Kubernetes cluster, or a single Docker container.
The port requirements differ based on the Rancher server architecture.
{{% tabs %}}
{{% tab "K3s" %}}
As of Rancher v2.5, Rancher can be installed on any Kubernetes cluster. For Rancher installs on a K3s, RKE, or RKE2 Kubernetes cluster, refer to the tabs below. For other Kubernetes distributions, refer to the distribution's documentation for the port requirements for cluster nodes.
> **Notes:**
>
> - Rancher nodes may also require additional outbound access for any external authentication provider which is configured (LDAP for example).
> - Kubernetes recommends TCP 30000-32767 for node port services.
> - For firewalls, traffic may need to be enabled within the cluster and pod CIDR.
### Ports for Rancher Server Nodes on K3s
{{% accordion label="Click to expand" %}}
The K3s server needs port 6443 to be accessible by the nodes.
@@ -23,24 +48,60 @@ If you wish to utilize the metrics server, you will need to open port 10250 on e
> **Important:** The VXLAN port on nodes should not be exposed to the world as it opens up your cluster network to be accessed by anyone. Run your nodes behind a firewall/security group that disables access to port 8472.
The following tables break down the port requirements for inbound and outbound traffic:
<figcaption>Inbound Rules for Rancher Server Nodes</figcaption>
| Protocol | Port | Source | Description
|-----|-----|----------------|---|
| TCP | 80 | Load balancer/proxy that does external SSL termination | Rancher UI/API when external SSL termination is used |
| TCP | 443 | <ul><li>server nodes</li><li>agent nodes</li><li>hosted/imported Kubernetes</li><li>any source that needs to be able to use the Rancher UI or API</li></ul> | Rancher agent, Rancher UI/API, kubectl |
| TCP | 6443 | K3s server nodes | Kubernetes API
| UDP | 8472 | K3s server and agent nodes | Required only for Flannel VXLAN.
| TCP | 10250 | K3s server and agent nodes | kubelet
Typically all outbound traffic is allowed.
<figcaption>Outbound Rules for Rancher Nodes</figcaption>
| Protocol | Port | Destination | Description |
| -------- | ---- | -------------------------------------------------------- | --------------------------------------------- |
| TCP | 22 | Any node IP from a node created using Node Driver | SSH provisioning of nodes using Node Driver |
| TCP | 443 | `35.160.43.145/32`, `35.167.242.46/32`, `52.33.59.17/32` | git.rancher.io (catalogs) |
| TCP | 2376 | Any node IP from a node created using Node driver | Docker daemon TLS port used by Docker Machine |
| TCP | 6443 | Hosted/Imported Kubernetes API | Kubernetes API server |
{{% /accordion %}}
### Ports for Rancher Server Nodes on RKE
{{% accordion label="Click to expand" %}}
Typically Rancher is installed on three RKE nodes that all have the etcd, control plane and worker roles.
The following tables break down the port requirements for traffic between the Rancher nodes:
<figcaption>Rules for traffic between Rancher nodes</figcaption>
| Protocol | Port | Description |
|-----|-----|----------------|
| TCP | 443 | Rancher agents |
| TCP | 2379 | etcd client requests |
| TCP | 2380 | etcd peer communication |
| TCP | 6443 | Kubernetes apiserver |
| UDP | 8472 | Canal/Flannel VXLAN overlay networking |
| TCP | 9099 | Canal/Flannel livenessProbe/readinessProbe |
| TCP | 10250 | kubelet |
| TCP | 10254 | Ingress controller livenessProbe/readinessProbe |
The following tables break down the port requirements for inbound and outbound traffic:
{{% /tab %}}
{{% tab "RKE" %}}
<figcaption>Inbound Rules for Rancher Nodes</figcaption>
| Protocol | Port | Source | Description |
|-----|-----|----------------|---|
| TCP | 22 | RKE CLI | SSH provisioning of node by RKE |
| TCP | 80 | Load Balancer/Reverse Proxy | HTTP traffic to Rancher UI/API |
| TCP | 443 | <ul><li>Load Balancer/Reverse Proxy</li><li>IPs of all cluster nodes and other API/UI clients</li></ul> | HTTPS traffic to Rancher UI/API |
| TCP | 6443 | Kubernetes API clients | HTTPS traffic to Kubernetes API |
<figcaption>Outbound Rules for Rancher Nodes</figcaption>
@@ -49,10 +110,45 @@ Typically all outbound traffic is allowed.
| TCP | 443 | `35.160.43.145`,`35.167.242.46`,`52.33.59.17` | Rancher catalog (git.rancher.io) |
| TCP | 22 | Any node created using a node driver | SSH provisioning of node by node driver |
| TCP | 2376 | Any node created using a node driver | Docker daemon TLS port used by node driver |
| TCP | 6443 | Hosted/Imported Kubernetes API | Kubernetes API server |
| TCP | Provider dependent | Port of the Kubernetes API endpoint in hosted cluster | Kubernetes API |
{{% /tab %}}
{{% tab "Docker" %}}
{{% /accordion %}}
### Ports for Rancher Server Nodes on RancherD or RKE2
{{% accordion label="Click to expand" %}}
The RancherD (or RKE2) server needs port 6443 and 9345 to be accessible by other nodes in the cluster.
All nodes need to be able to reach other nodes over UDP port 8472 when Flannel VXLAN is used.
If you wish to utilize the metrics server, you will need to open port 10250 on each node.
**Important:** The VXLAN port on nodes should not be exposed to the world as it opens up your cluster network to be accessed by anyone. Run your nodes behind a firewall/security group that disables access to port 8472.
<figcaption>Inbound Rules for RancherD or RKE2 Server Nodes</figcaption>
| Protocol | Port | Source | Description
|-----|-----|----------------|---|
| TCP | 9345 | RancherD/RKE2 agent nodes | Kubernetes API
| TCP | 6443 | RancherD/RKE2 agent nodes | Kubernetes API
| UDP | 8472 | RancherD/RKE2 server and agent nodes | Required only for Flannel VXLAN
| TCP | 10250 | RancherD/RKE2 server and agent nodes | kubelet
| TCP | 2379 | RancherD/RKE2 server nodes | etcd client port
| TCP | 2380 | RancherD/RKE2 server nodes | etcd peer port
| TCP | 30000-32767 | RancherD/RKE2 server and agent nodes | NodePort port range
| HTTP | 8080 | Load balancer/proxy that does external SSL termination | Rancher UI/API when external SSL termination is used |
| HTTPS | 8443 | <ul><li>hosted/imported Kubernetes</li><li>any source that needs to be able to use the Rancher UI or API</li></ul> | Rancher agent, Rancher UI/API, kubectl. Not needed if you have LB doing TLS termination. |
Typically all outbound traffic is allowed.
{{% /accordion %}}
### Ports for Rancher Server in Docker
{{% accordion label="Click to expand" %}}
The following tables break down the port requirements for Rancher nodes, for inbound and outbound traffic:
<figcaption>Inbound Rules for Rancher Node</figcaption>
@@ -70,28 +166,27 @@ Typically all outbound traffic is allowed.
| TCP | 2376 | Any node IP from a node created using a node driver | Docker daemon TLS port used by Docker Machine |
| TCP | 6443 | Hosted/Imported Kubernetes API | Kubernetes API server |
{{% /tab %}}
{{% /tabs %}}
{{% /accordion %}}
> **Notes:**
>
> - Rancher nodes may also require additional outbound access for any external authentication provider which is configured (LDAP for example).
> - Kubernetes recommends TCP 30000-32767 for node port services.
> - For firewalls, traffic may need to be enabled within the cluster and pod CIDR.
## Downstream Kubernetes Cluster Nodes
# Downstream Kubernetes Cluster Nodes
Downstream Kubernetes clusters run your apps and services. This section describes what ports need to be opened on the nodes in downstream clusters so that Rancher can communicate with them.
The port requirements differ depending on how the downstream cluster was launched. Each of the tabs below list the ports that need to be opened for different [cluster types]({{<baseurl>}}/rancher/v2.x/en/cluster-provisioning/#cluster-creation-options).
The following diagram depicts the ports that are opened for each [cluster type]({{<baseurl>}}/rancher/v2.x/en/cluster-provisioning).
<figcaption>Port Requirements for the Rancher Management Plane</figcaption>
![Basic Port Requirements]({{<baseurl>}}/img/rancher/port-communications.svg)
>**Tip:**
>
>If security isn't a large concern and you're okay with opening a few additional ports, you can use the table in [Commonly Used Ports](#commonly-used-ports) as your port reference instead of the comprehensive tables below.
{{% tabs %}}
### Ports for Rancher Launched Kubernetes Clusters using Node Pools
{{% tab "Node Pools" %}}
{{% accordion label="Click to expand" %}}
The following table depicts the port requirements for [Rancher Launched Kubernetes]({{<baseurl>}}/rancher/v2.x/en/cluster-provisioning/rke-clusters/) with nodes created in an [Infrastructure Provider]({{<baseurl>}}/rancher/v2.x/en/cluster-provisioning/rke-clusters/node-pools/).
@@ -100,56 +195,48 @@ The following table depicts the port requirements for [Rancher Launched Kubernet
{{< ports-iaas-nodes >}}
{{% /tab %}}
{{% /accordion %}}
{{% tab "Custom Nodes" %}}
### Ports for Rancher Launched Kubernetes Clusters using Custom Nodes
{{% accordion label="Click to expand" %}}
The following table depicts the port requirements for [Rancher Launched Kubernetes]({{<baseurl>}}/rancher/v2.x/en/cluster-provisioning/rke-clusters/) with [Custom Nodes]({{<baseurl>}}/rancher/v2.x/en/cluster-provisioning/rke-clusters/custom-nodes/).
{{< ports-custom-nodes >}}
{{% /tab %}}
{{% /accordion %}}
{{% tab "Hosted Clusters" %}}
### Ports for Hosted Kubernetes Clusters
{{% accordion label="Click to expand" %}}
The following table depicts the port requirements for [hosted clusters]({{<baseurl>}}/rancher/v2.x/en/cluster-provisioning/hosted-kubernetes-clusters).
{{< ports-imported-hosted >}}
{{% /tab %}}
{{% /accordion %}}
{{% tab "Imported Clusters" %}}
### Ports for Registered Clusters
Note: Registered clusters were called imported clusters prior to Rancher v2.5.
{{% accordion label="Click to expand" %}}
The following table depicts the port requirements for [imported clusters]({{<baseurl>}}/rancher/v2.x/en/cluster-provisioning/imported-clusters/).
{{< ports-imported-hosted >}}
{{% /tab %}}
{{% /tabs %}}
{{% /accordion %}}
## Other Port Considerations
# Other Port Considerations
### Commonly Used Ports
These ports are typically opened on your Kubernetes nodes, regardless of what type of cluster it is.
| Protocol | Port | Description |
|:--------: |:----------------: |------------------------------------------------- |
| TCP | 22 | Node driver SSH provisioning |
| TCP | 2376 | Node driver Docker daemon TLS port |
| TCP | 2379 | etcd client requests |
| TCP | 2380 | etcd peer communication |
| UDP | 8472 | Canal/Flannel VXLAN overlay networking |
| UDP | 4789 | Flannel VXLAN overlay networking on Windows cluster |
| TCP | 9099 | Canal/Flannel livenessProbe/readinessProbe |
| TCP | 9796 | Default port required by Monitoring to scrape metrics |
| TCP | 6783 | Weave Port |
| UDP | 6783-6784 | Weave UDP Ports |
| TCP | 10250 | kubelet API |
| TCP | 10254 | Ingress controller livenessProbe/readinessProbe |
| TCP/UDP | 30000-</br>32767 | NodePort port range |
{{% include file="/rancher/v2.x/en/installation/requirements/ports/common-ports-table" %}}
----
@@ -165,7 +252,7 @@ However, this traffic may be blocked when:
In these cases, you have to explicitly allow this traffic in your host firewall, or in case of public/private cloud hosted machines (i.e. AWS or OpenStack), in your security group configuration. Keep in mind that when using a security group as source or destination in your security group, explicitly opening ports only applies to the private interface of the nodes / instances.
### Rancher AWS EC2 security group
### Rancher AWS EC2 Security Group
When using the [AWS EC2 node driver]({{<baseurl>}}/rancher/v2.x/en/cluster-provisioning/rke-clusters/node-pools/ec2/) to provision cluster nodes in Rancher, you can choose to let Rancher create a security group called `rancher-nodes`. The following rules are automatically added to this security group.
@@ -181,6 +268,24 @@ When using the [AWS EC2 node driver]({{<baseurl>}}/rancher/v2.x/en/cluster-provi
| Custom UDP Rule | UDP | 8472 | sg-xxx (rancher-nodes) | Inbound |
| Custom TCP Rule | TCP | 10250-10252 | sg-xxx (rancher-nodes) | Inbound |
| Custom TCP Rule | TCP | 10256 | sg-xxx (rancher-nodes) | Inbound |
| Custom TCP Rule | TCP | 30000-32767 | 0.0.0.0/0 | Inbound |
| Custom UDP Rule | UDP | 30000-32767 | 0.0.0.0/0 | Inbound |
| Custom TCP Rule | TCP | 30000-32767 | 0.0.0.0/0 | Inbound |
| Custom UDP Rule | UDP | 30000-32767 | 0.0.0.0/0 | Inbound |
| All traffic | All | All | 0.0.0.0/0 | Outbound |
### Opening SUSE Linux Ports
SUSE Linux may have a firewall that blocks all ports by default. To open the ports needed for adding the host to a custom cluster,
1. SSH into the instance.
1. Edit /`etc/sysconfig/SuSEfirewall2` and open the required ports. In this example, ports 9796 and 10250 are also opened for monitoring:
```
FW_SERVICES_EXT_TCP="22 80 443 2376 2379 2380 6443 9099 9796 10250 10254 30000:32767"
FW_SERVICES_EXT_UDP="8472 30000:32767"
FW_ROUTE=yes
```
1. Restart the firewall with the new ports:
```
SuSEfirewall2
```
**Result:** The node has the open ports required to be added to a custom cluster.
content/rancher/v2.x/en/installation/requirements/ports/common-ports-table/index.md
+22
View File
@@ -0,0 +1,22 @@
---
headless: true
---
| Protocol | Port | Description |
|:--------: |:----------------: |---------------------------------------------------------------------------------- |
| TCP | 22 | Node driver SSH provisioning |
| TCP | 179 | Calico BGP Port |
| TCP | 2376 | Node driver Docker daemon TLS port |
| TCP | 2379 | etcd client requests |
| TCP | 2380 | etcd peer communication |
| UDP | 8472 | Canal/Flannel VXLAN overlay networking |
| UDP | 4789 | Flannel VXLAN overlay networking on Windows cluster |
| TCP | 8443 | Rancher webhook |
| TCP | 9099 | Canal/Flannel livenessProbe/readinessProbe |
| TCP | 9100 | Default port required by Monitoring to scrape metrics from Linux node-exporters |
| TCP | 9443 | Rancher webhook |
| TCP | 9796 | Default port required by Monitoring to scrape metrics from Windows node-exporters |
| TCP | 6783 | Weave Port |
| UDP | 6783-6784 | Weave UDP Ports |
| TCP | 10250 | kubelet API |
| TCP | 10254 | Ingress controller livenessProbe/readinessProbe |
| TCP/UDP | 30000-</br>32767 | NodePort port range |
content/rancher/v2.x/en/installation/resources/_index.md
+1 -1
View File
@@ -1,6 +1,6 @@
---
title: Resources
weight: 4
weight: 5
aliases:
- /rancher/v2.x/en/installation/options
---
content/rancher/v2.x/en/installation/resources/advanced/firewall/_index.md
+1 -1
View File
@@ -105,4 +105,4 @@ After the `firewall-cmd` commands have been run on a node, use the following com
firewall-cmd --reload
```
**Result:** The firewall is updated so that Helm can communicate with the Rancher server nodes.
**Result:** The firewall is updated so that Helm can communicate with the Rancher server nodes.
content/rancher/v2.x/en/installation/resources/encryption/custom-ca-root-certificate/_index.md → content/rancher/v2.x/en/installation/resources/custom-ca-root-certificate/_index.md
+3 -2
View File
@@ -3,6 +3,7 @@ title: About Custom CA Root Certificates
weight: 1
aliases:
- /rancher/v2.x/en/installation/options/custom-ca-root-certificate/
- /rancher/v2.x/en/installation/resources/choosing-version/encryption/custom-ca-root-certificate
---
If you're using Rancher in an internal production environment where you aren't exposing apps publicly, use a certificate from a private certificate authority (CA).
@@ -21,7 +22,7 @@ Examples of services that Rancher can access:
For details on starting a Rancher container with your private CA certificates mounted, refer to the installation docs:
- [Docker Install]({{<baseurl>}}/rancher/v2.x/en/installation/other-installation-methods/single-node-docker/advanced/#custom-ca-certificate)
- [Docker install Custom CA certificate options]({{<baseurl>}}/rancher/v2.x/en/installation/other-installation-methods/single-node-docker/advanced/#custom-ca-certificate)
- [Kubernetes Install]({{<baseurl>}}/rancher/v2.x/en/installation/resources/chart-options/#additional-trusted-cas)
- [Kubernetes install options for Additional Trusted CAs]({{<baseurl>}}/rancher/v2.x/en/installation/install-rancher-on-k8s/chart-options/#additional-trusted-cas)
content/rancher/v2.x/en/installation/resources/encryption/_index.md
-6
View File
@@ -1,6 +0,0 @@
---
title: Encryption
weight: 3
---
The documents in this section contain information about certificate configuration and `cert-manager`.
content/rancher/v2.x/en/installation/resources/encryption/tls-secrets/_index.md → content/rancher/v2.x/en/installation/resources/tls-secrets/_index.md
+1
View File
@@ -3,6 +3,7 @@ title: Adding TLS Secrets
weight: 2
aliases:
- /rancher/v2.x/en/installation/options/tls-secrets/
- /rancher/v2.x/en/installation/resources/encryption/tls-secrets
---
Kubernetes will create all the objects and services for Rancher, but it will not become available until we populate the `tls-rancher-ingress` secret in the `cattle-system` namespace with the certificate and key.
content/rancher/v2.x/en/installation/resources/encryption/tls-settings/_index.md → content/rancher/v2.x/en/installation/resources/tls-settings/_index.md
+6 -5
View File
@@ -4,19 +4,20 @@ weight: 3
aliases:
- /rancher/v2.x/en/installation/options/tls-settings/
- /rancher/v2.x/en/admin-settings/tls-settings
- /rancher/v2.x/en/installation/resources/encryption/tls-settings
---
In Rancher v2.1.7, the default TLS configuration changed to only accept TLS 1.2 and secure TLS cipher suites. TLS 1.3 and TLS 1.3 exclusive cipher suites are not supported.
## Configuring TLS settings
# Configuring TLS settings
The Audit Log is enabled and configured by passing environment variables to the Rancher server container. See the following to enable on your installation.
- [Installing Rancher on a single node with Docker]({{<baseurl>}}/rancher/v2.x/en/installation/other-installation-methods/single-node-docker/advanced/#tls-settings)
- [TLS settings in Docker options]({{<baseurl>}}/v2.x/en/installation/other-installation-methods/single-node-docker/advanced/#tls-settings)
- [Installing Rancher on Kubernetes]({{<baseurl>}}/rancher/v2.x/en/installation/resources/chart-options/#tls-settings)
- [TLS settings in Helm chart options]({{<baseurl>}}/rancher/v2.x/en/installation/install-rancher-on-k8s/chart-options/#tls-settings)
## TLS settings
# TLS Environment Variables
| Parameter | Description | Default | Available options |
|-----|-----|-----|-----|
@@ -24,7 +25,7 @@ The Audit Log is enabled and configured by passing environment variables to the
| `CATTLE_TLS_CIPHERS` | Allowed TLS cipher suites | `TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,`<br/>`TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,`<br/>`TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,`<br/>`TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,`<br/>`TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,`<br/>`TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305` | See [Golang tls constants](https://golang.org/pkg/crypto/tls/#pkg-constants) |
## Legacy configuration
# Legacy configuration
If you need to configure TLS the same way as it was before Rancher v2.1.7, please use the following settings:
content/rancher/v2.x/en/installation/resources/encryption/upgrading-cert-manager/_index.md → content/rancher/v2.x/en/installation/resources/upgrading-cert-manager/_index.md
+1
View File
@@ -4,6 +4,7 @@ weight: 4
aliases:
- /rancher/v2.x/en/installation/options/upgrading-cert-manager
- /rancher/v2.x/en/installation/options/upgrading-cert-manager/helm-2-instructions
- /rancher/v2.x/en/installation/resources/encryption/upgrading-cert-manager
---
Rancher uses cert-manager to automatically generate and renew TLS certificates for HA deployments of Rancher. As of Fall 2019, three important changes to cert-manager are set to occur that you need to take action on if you have an HA deployment of Rancher:
content/rancher/v2.x/en/installation/resources/encryption/upgrading-cert-manager/helm-2-instructions/_index.md → content/rancher/v2.x/en/installation/resources/upgrading-cert-manager/helm-2-instructions/_index.md
+1
View File
@@ -3,6 +3,7 @@ title: Upgrading Cert-Manager with Helm 2
weight: 2040
aliases:
- /rancher/v2.x/en/installation/options/upgrading-cert-manager/helm-2-instructions
- /rancher/v2.x/en/installation/resources/choosing-version/encryption/upgrading-cert-manager/helm-2-instructions
---
Rancher uses cert-manager to automatically generate and renew TLS certificates for HA deployments of Rancher. As of Fall 2019, three important changes to cert-manager are set to occur that you need to take action on if you have an HA deployment of Rancher:
content/rancher/v2.x/en/installation/upgrades-rollbacks/_index.md
-19
View File
@@ -1,19 +0,0 @@
---
title: Upgrades and Rollbacks
weight: 150
aliases:
- /rancher/v2.x/en/upgrades
---
### Upgrading Rancher
To upgrade Rancher, refer to [these instructions.](./upgrades/)
### Rolling Back Unsuccessful Upgrades
In the event that your Rancher Server does not upgrade successfully, you can rollback to your installation prior to upgrade using these instructions: [Rollbacks for Rancher installed on a Kubernetes cluster](./rollbacks/ha-server-rollbacks)
> **Note:** If you are rolling back to versions in either of these scenarios, you must follow some extra [instructions]({{<baseurl>}}/rancher/v2.x/en/upgrades/rollbacks/) in order to get your clusters working.
>
>- Rolling back from v2.1.6+ to any version between v2.1.0 - v2.1.5 or v2.0.0 - v2.0.10.
>- Rolling back from v2.0.11+ to any version between v2.0.0 - v2.0.10.
content/rancher/v2.x/en/installation/upgrades-rollbacks/rollbacks/ha-server-rollbacks/_index.md
-17
View File
@@ -1,17 +0,0 @@
---
title: Kubernetes Rollback
weight: 1025
aliases:
- /rancher/v2.x/en/upgrades/ha-server-rollbacks
- /rancher/v2.x/en/upgrades/rollbacks/ha-server-rollbacks
---
If you upgrade Rancher and the upgrade does not complete successfully, you may need to rollback your Rancher Server to its last healthy state.
To restore Rancher prior to v2.5, follow the procedure detailed here: [Restoring Backups — Kubernetes installs]({{<baseurl>}}/rancher/v2.x/en/backups/restorations/ha-restoration)
To restore Rancher v2.5, you can use the `rancher-backup` application and restore Rancher from backup according to [this section.]({{<baseurl>}}/rancher/v2.x/en/backups/restoring-rancher/)
Restoring a snapshot of the Rancher Server cluster will revert Rancher to the version and state at the time of the snapshot.
>**Note:** Managed cluster are authoritative for their state. This means restoring the rancher server will not revert workload deployments or changes made on managed clusters after the snapshot was taken.
content/rancher/v2.x/en/installation/upgrades-rollbacks/upgrades/_index.md
-33
View File
@@ -1,33 +0,0 @@
---
title: Upgrades
weight: 1005
aliases:
- /rancher/v2.x/en/upgrades/upgrades
---
This section contains information about how to upgrade your Rancher server to a newer version. Regardless if you installed in an air gap environment or not, the upgrade steps mainly depend on whether you have a single node or high-availability installation of Rancher. Select from the following options:
- [Upgrading Rancher installed with Docker]({{<baseurl>}}/rancher/v2.x/en/installation/other-installation-methods/single-node-docker/single-node-upgrades)
- [Upgrading Rancher installed on a Kubernetes cluster]({{<baseurl>}}/rancher/v2.x/en/installation/upgrades-rollbacks/upgrades/ha/)
### Known Upgrade Issues
The following table lists some of the most noteworthy issues to be considered when upgrading Rancher. A more complete list of known issues for each Rancher version can be found in the release notes on [GitHub](https://github.com/rancher/rancher/releases) and on the [Rancher forums.](https://forums.rancher.com/c/announcements/12)
Upgrade Scenario | Issue
---|---
Upgrading to v2.4.6 or v2.4.7 | These Rancher versions had an issue where the `kms:ListKeys` permission was required to create, edit, or clone Amazon EC2 node templates. This requirement was removed in v2.4.8.
Upgrading to v2.3.0+ | Any user provisioned cluster will be automatically updated upon any edit as tolerations were added to the images used for Kubernetes provisioning.
Upgrading to v2.2.0-v2.2.x | Rancher introduced the [system charts](https://github.com/rancher/system-charts) repository which contains all the catalog items required for features such as monitoring, logging, alerting and global DNS. To be able to use these features in an air gap install, you will need to mirror the `system-charts` repository locally and configure Rancher to use that repository. Please follow the instructions to [configure Rancher system charts]({{<baseurl>}}/rancher/v2.x/en/installation/resources/local-system-charts/#setting-up-system-charts-for-rancher-prior-to-v2-3-0).
Upgrading from v2.0.13 or earlier | If your cluster's certificates have expired, you will need to perform [additional steps]({{<baseurl>}}/rancher/v2.x/en/cluster-admin/certificate-rotation/#rotating-expired-certificates-after-upgrading-older-rancher-versions) to rotate the certificates.
Upgrading from v2.0.7 or earlier | Rancher introduced the `system` project, which is a project that's automatically created to store important namespaces that Kubernetes needs to operate. During upgrade to v2.0.7+, Rancher expects these namespaces to be unassigned from all projects. Before beginning upgrade, check your system namespaces to make sure that they're unassigned to [prevent cluster networking issues]({{<baseurl>}}/rancher/v2.x/en/upgrades/upgrades/namespace-migration/#preventing-cluster-networking-issues).
### Caveats
Upgrades _to_ or _from_ any chart in the [rancher-alpha repository]({{<baseurl>}}/rancher/v2.x/en/installation/resources/chart-options/#helm-chart-repositories/) aren't supported.
### RKE Add-on Installs
**Important: RKE add-on install is only supported up to Rancher v2.0.8**
Please use the Rancher helm chart to install Rancher on a Kubernetes cluster. For details, see the [Kubernetes Install - Installation Outline]({{<baseurl>}}/rancher/v2.x/en/installation/install-rancher-on-k8s/#installation-outline).
If you are currently using the RKE add-on install method, see [Migrating from a RKE add-on install]({{<baseurl>}}/rancher/v2.x/en/upgrades/upgrades/migrating-from-rke-add-on/) for details on how to move to using the helm chart.
content/rancher/v2.x/en/installation/upgrades-rollbacks/upgrades/ha/_index.md
-231
View File
@@ -1,231 +0,0 @@
---
title: Upgrading Rancher Installed on Kubernetes
weight: 1020
aliases:
- /rancher/v2.x/en/upgrades/upgrades/ha-server-upgrade-helm-airgap
- /rancher/v2.x/en/upgrades/air-gap-upgrade/
- /rancher/v2.x/en/upgrades/upgrades/ha
- /rancher/v2.x/en/installation/install-rancher-on-k8s/upgrades/upgrades/ha
- /rancher/v2.x/en/installation/upgrades-rollbacks/upgrades/
- /rancher/v2.x/en/upgrades/upgrades/ha-server-upgrade-helm/
---
The following instructions will guide you through using Helm to upgrade a Rancher server that was installed on a Kubernetes cluster.
To upgrade the components in your Kubernetes cluster, or the definition of the [Kubernetes services]({{<baseurl>}}/rke/latest/en/config-options/services/) or [add-ons]({{<baseurl>}}/rke/latest/en/config-options/add-ons/), refer to the [upgrade documentation for RKE]({{<baseurl>}}/rke/latest/en/upgrades/), the Rancher Kubernetes Engine.
If you installed Rancher using the RKE Add-on yaml, follow the directions to [migrate or upgrade]({{<baseurl>}}/rancher/v2.x/en/upgrades/upgrades/migrating-from-rke-add-on).
>**Notes:**
>
> - If you are upgrading to Rancher v2.5 from a Rancher server that was started with the Helm chart option `--add-local=false`, you will need to drop that flag when upgrading. Otherwise, the Rancher server will not start. The `restricted-admin` role can be used to continue restricting access to the local cluster. For more information, see [this section.]({{<baseurl>}}/rancher/v2.x/en/admin-settings/rbac/global-permissions/#upgrading-from-rancher-with-a-hidden-local-cluster)
> - [Let's Encrypt will be blocking cert-manager instances older than 0.8.0 starting November 1st 2019.](https://community.letsencrypt.org/t/blocking-old-cert-manager-versions/98753) Upgrade cert-manager to the latest version by following [these instructions.]({{<baseurl>}}/rancher/v2.x/en/installation/options/upgrading-cert-manager)
> - Helm should be run from the same location as your kubeconfig file (where you run your kubectl commands from. If you installed K8s with RKE, the config will have been created in the directory you ran `rke up` in) or should manually target the kubeconfig for the intended cluster with the `--kubeconfig` tag (see: https://helm.sh/docs/helm/helm/)
> - The upgrade instructions assume you are using Helm 3. For migration of installs started with Helm 2, refer to the official [Helm 2 to 3 migration docs.](https://helm.sh/blog/migrate-from-helm-v2-to-helm-v3/) The [Helm 2 upgrade page here]({{<baseurl>}}/rancher/v2.x/en/installation/upgrades-rollbacks/upgrades/ha/helm2)provides a copy of the older upgrade instructions that used Helm 2, and it is intended to be used if upgrading to Helm 3 is not feasible.
> - If you are upgrading Rancher from v2.x to v2.3+, and you are using external TLS termination, you will need to edit the cluster.yml to [enable using forwarded host headers.]({{<baseurl>}}/rancher/v2.x/en/installation/resources/chart-options/#configuring-ingress-for-external-tls-when-using-nginx-v0-25)
# Prerequisites
- **Review the [known upgrade issues]({{<baseurl>}}/rancher/v2.x/en/upgrades/upgrades/#known-upgrade-issues) and [caveats]({{<baseurl>}}/rancher/v2.x/en/upgrades/upgrades/#caveats)** in the Rancher documentation for the most noteworthy issues to consider when upgrading Rancher. A more complete list of known issues for each Rancher version can be found in the release notes on [GitHub](https://github.com/rancher/rancher/releases) and on the [Rancher forums.](https://forums.rancher.com/c/announcements/12)
- **For [air gap installs only,]({{<baseurl>}}/rancher/v2.x/en/installation/other-installation-methods/air-gap) collect and populate images for the new Rancher server version.** Follow the guide to [populate your private registry]({{<baseurl>}}/rancher/v2.x/en/installation/other-installation-methods/air-gap/populate-private-registry/) with the images for the Rancher version that you want to upgrade to.
# Upgrade Outline
Follow the steps to upgrade Rancher server:
- [A. Back up your Kubernetes cluster that is running Rancher server](#a-back-up-your-kubernetes-cluster-that-is-running-rancher-server)
- [B. Update the Helm chart repository](#b-update-the-helm-chart-repository)
- [C. Upgrade Rancher](#c-upgrade-rancher)
- [D. Verify the Upgrade](#d-verify-the-upgrade)
### A. Back up Your Kubernetes Cluster that is Running Rancher Server
[Take a one-time snapshot]({{<baseurl>}}/rancher/v2.x/en/backups/backups/ha-backups/#option-b-one-time-snapshots)
of your Kubernetes cluster running Rancher server. You'll use the snapshot as a restoration point if something goes wrong during upgrade.
### B. Update the Helm chart repository
1. Update your local helm repo cache.
```
helm repo update
```
1. Get the repository name that you used to install Rancher.
For information about the repos and their differences, see [Helm Chart Repositories]({{<baseurl>}}/rancher/v2.x/en/installation/resources/chart-options/#helm-chart-repositories).
{{< release-channel >}}
```
helm repo list
NAME URL
stable https://kubernetes-charts.storage.googleapis.com
rancher-<CHART_REPO> https://releases.rancher.com/server-charts/<CHART_REPO>
```
> **Note:** If you want to switch to a different Helm chart repository, please follow the [steps on how to switch repositories]({{<baseurl>}}/rancher/v2.x/en/installation/resources/chart-options/#switching-to-a-different-helm-chart-repository). If you switch repositories, make sure to list the repositories again before continuing onto Step 3 to ensure you have the correct one added.
1. Fetch the latest chart to install Rancher from the Helm chart repository.
This command will pull down the latest charts and save it in the current directory as a `.tgz` file.
```plain
helm fetch rancher-<CHART_REPO>/rancher
```
You can fetch the chart for the specific version you are upgrading to by adding in the `--version=` tag. For example:
```plain
helm fetch rancher-<CHART_REPO>/rancher --version=v2.4.11
```
### C. Upgrade Rancher
This section describes how to upgrade normal (Internet-connected) or air gap installations of Rancher with Helm.
{{% tabs %}}
{{% tab "Kubernetes Upgrade" %}}
Get the values, which were passed with `--set`, from the current Rancher Helm chart that is installed.
```
helm get values rancher -n cattle-system
hostname: rancher.my.org
```
> **Note:** There will be more values that are listed with this command. This is just an example of one of the values.
If you are also upgrading cert-manager to the latest version from a version older than 0.11.0, follow `Option B: Reinstalling Rancher and cert-manager`. Otherwise, follow `Option A: Upgrading Rancher`.
{{% accordion label="Option A: Upgrading Rancher" %}}
Upgrade Rancher to the latest version with all your settings.
Take all the values from the previous step and append them to the command using `--set key=value`:
```
helm upgrade rancher rancher-<CHART_REPO>/rancher \
--namespace cattle-system \
--set hostname=rancher.my.org
```
> **Note:** The above is an example, there may be more values from the previous step that need to be appended.
Alternatively, it's possible to export the current values to a file and reference that file during upgrade. For example, to only change the Rancher version:
```
helm get values rancher -n cattle-system -o yaml > values.yaml
helm upgrade rancher rancher-<CHART_REPO>/rancher \
--namespace cattle-system \
-f values.yaml \
--version=2.4.5
```
{{% /accordion %}}
{{% accordion label="Option B: Reinstalling Rancher and cert-manager" %}}
If you are currently running the cert-manger whose version is older than v0.11, and want to upgrade both Rancher and cert-manager to a newer version, then you need to reinstall both Rancher and cert-manger due to the API change in cert-manger v0.11.
1. Uninstall Rancher
```
helm delete rancher -n cattle-system
```
2. Uninstall and reinstall `cert-manager` according to the instructions on the [Upgrading Cert-Manager]({{<baseurl>}}/rancher/v2.x/en/installation/options/upgrading-cert-manager) page.
3. Reinstall Rancher to the latest version with all your settings. Take all the values from the step 1 and append them to the command using `--set key=value`. Note: There will be many more options from the step 1 that need to be appended.
```
helm install rancher rancher-<CHART_REPO>/rancher \
--namespace cattle-system \
--set hostname=rancher.my.org
```
{{% /accordion %}}
{{% /tab %}}
{{% tab "Kubernetes Air Gap Upgrade" %}}
1. Render the Rancher template using the same chosen options that were used when installing Rancher. Use the reference table below to replace each placeholder. Rancher needs to be configured to use the private registry in order to provision any Rancher launched Kubernetes clusters or Rancher tools.
Based on the choice you made during installation, complete one of the procedures below.
Placeholder | Description
------------|-------------
`<VERSION>` | The version number of the output tarball.
`<RANCHER.YOURDOMAIN.COM>` | The DNS name you pointed at your load balancer.
`<REGISTRY.YOURDOMAIN.COM:PORT>` | The DNS name for your private registry.
`<CERTMANAGER_VERSION>` | Cert-manager version running on k8s cluster.
{{% accordion id="self-signed" label="Option A-Default Self-Signed Certificate" %}}
```plain
helm template ./rancher-<VERSION>.tgz --output-dir . \
--name rancher \
--namespace cattle-system \
--set hostname=<RANCHER.YOURDOMAIN.COM> \
--set certmanager.version=<CERTMANAGER_VERSION> \
--set rancherImage=<REGISTRY.YOURDOMAIN.COM:PORT>/rancher/rancher \
--set systemDefaultRegistry=<REGISTRY.YOURDOMAIN.COM:PORT> \ # Available as of v2.2.0, set a default private registry to be used in Rancher
--set useBundledSystemChart=true # Available as of v2.3.0, use the packaged Rancher system charts
```
{{% /accordion %}}
{{% accordion id="secret" label="Option B: Certificates From Files using Kubernetes Secrets" %}}
```plain
helm template ./rancher-<VERSION>.tgz --output-dir . \
--name rancher \
--namespace cattle-system \
--set hostname=<RANCHER.YOURDOMAIN.COM> \
--set rancherImage=<REGISTRY.YOURDOMAIN.COM:PORT>/rancher/rancher \
--set ingress.tls.source=secret \
--set systemDefaultRegistry=<REGISTRY.YOURDOMAIN.COM:PORT> \ # Available as of v2.2.0, set a default private registry to be used in Rancher
--set useBundledSystemChart=true # Available as of v2.3.0, use the packaged Rancher system charts
```
If you are using a Private CA signed cert, add `--set privateCA=true` following `--set ingress.tls.source=secret`:
```plain
helm template ./rancher-<VERSION>.tgz --output-dir . \
--name rancher \
--namespace cattle-system \
--set hostname=<RANCHER.YOURDOMAIN.COM> \
--set rancherImage=<REGISTRY.YOURDOMAIN.COM:PORT>/rancher/rancher \
--set ingress.tls.source=secret \
--set privateCA=true \
--set systemDefaultRegistry=<REGISTRY.YOURDOMAIN.COM:PORT> \ # Available as of v2.2.0, set a default private registry to be used in Rancher
--set useBundledSystemChart=true # Available as of v2.3.0, use the packaged Rancher system charts
```
{{% /accordion %}}
2. Copy the rendered manifest directories to a system with access to the Rancher server cluster and apply the rendered templates.
Use `kubectl` to apply the rendered manifests.
```plain
kubectl -n cattle-system apply -R -f ./rancher
```
{{% /tab %}}
{{% /tabs %}}
### D. Verify the Upgrade
Log into Rancher to confirm that the upgrade succeeded.
>**Having network issues following upgrade?**
>
> See [Restoring Cluster Networking]({{<baseurl>}}/rancher/v2.x/en/upgrades/upgrades/namespace-migration/#restoring-cluster-networking).
## Rolling Back
Should something go wrong, follow the [roll back]({{<baseurl>}}/rancher/v2.x/en/upgrades/rollbacks/ha-server-rollbacks/) instructions to restore the snapshot you took before you preformed the upgrade.
content/rancher/v2.x/en/istio/v2.5/_index.md
+57 -4
View File
@@ -18,19 +18,17 @@ This core service mesh provides features that include but are not limited to the
After [setting up istio]({{<baseurl>}}/rancher/v2.x/en/cluster-admin/tools/istio/setup) you can leverage Istio's control plane functionality through the Cluster Explorer, `kubectl`, or `istioctl`.
Rancher's Istio integration comes with a comprehensive visualization aid:
- **Get the full picture of your microservice architecture with Kiali.** [Kiali](https://www.kiali.io/) provides a diagram that shows the services within a service mesh and how they are connected, including the traffic rates and latencies between them. You can check the health of the service mesh, or drill down to see the incoming and outgoing requests to a single component.
Istio needs to be set up by a `cluster-admin` before it can be used in a project.
- [What's New in Rancher v2.5](#what-s-new-in-rancher-v2-5)
- [Tools Bundled with Istio](#tools-bundled-with-istio)
- [Prerequisites](#prerequisites)
- [Setup Guide](#setup-guide)
- [Remove Istio](#remove-istio)
- [Migrate from Previous Istio Version](#migrate-from-previous-istio-version)
- [Accessing Visualizations](#accessing-visualizations)
- [Architecture](#architecture)
- [Additional steps for installing Istio on an RKE2 cluster](#additional-steps-for-installing-istio-on-an-rke2-cluster)
# What's New in Rancher v2.5
@@ -44,10 +42,32 @@ Istio has migrated away from Helm as a way to install Istio and now provides ins
This Helm chart will be available via the Apps and Marketplace in the UI. A user that has access to the Rancher Chart's catalog will need to set up Istio before it can be used in the project.
# Tools Bundled with Istio
Our [Istio](https://istio.io/) installer wraps the istioctl binary commands in a handy Helm chart, including an overlay file option to allow complex customization.
It also includes the following:
### Kiali
Kiali is a comprehensive visualization aid used for graphing traffic flow throughout the service mesh. It allows you to see how they are connected, including the traffic rates and latencies between them.
You can check the health of the service mesh, or drill down to see the incoming and outgoing requests to a single component.
### Jaeger
_Bundled as of v2.5.4_
Our Istio installer includes a quick-start, all-in-one installation of [Jaeger,](https://www.jaegertracing.io/) a tool used for tracing distributed systems.
Note that this is not a production-qualified deployment of Jaeger. This deployment uses an in-memory storage component, while a persistent storage component is recommended for production. For more information on which deployment strategy you may need, refer to the [Jaeger documentation.](https://www.jaegertracing.io/docs/latest/operator/#production-strategy)
# Prerequisites
Before enabling Istio, we recommend that you confirm that your Rancher worker nodes have enough [CPU and memory]({{<baseurl>}}/rancher/v2.x/en/cluster-admin/tools/istio/resources) to run all of the components of Istio.
If you are installing Istio on RKE2 cluster, some additional steps are required. For details, see [this section.](#additional-steps-for-installing-istio-on-an-rke2-cluster)
# Setup Guide
Refer to the [setup guide]({{<baseurl>}}/rancher/v2.x/en/cluster-admin/tools/istio/setup) for instructions on how to set up Istio and use it in a project.
@@ -99,3 +119,36 @@ By default, each Rancher-provisioned cluster has one NGINX ingress controller al
### Egress Support
By default the Egress gateway is disabled, but can be enabled on install or upgrade through the values.yaml or via the [overlay file]({{<baseurl>}}/rancher/v2.x/en/istio/setup/enable-istio-in-cluster/#overlay-file).
# Additional Steps for Installing Istio on an RKE2 Cluster
Through the **Cluster Explorer,** when installing or upgrading Istio through **Apps & Marketplace,**
1. Click **Components.**
1. Check the box next to **Enabled CNI.**
1. Add a custom overlay file specifying `cniBinDir` and `cniConfDir`. For more information on these options, refer to the [Istio documentation.](https://istio.io/latest/docs/setup/additional-setup/cni/#helm-chart-parameters) An example is below:
```yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
components:
cni:
enabled: true
values:
cni:
image: rancher/istio-install-cni:1.7.3
excludeNamespaces:
- istio-system
- kube-system
logLevel: info
cniBinDir: /opt/cni/bin
cniConfDir: /etc/cni/net.d
```
1. After installing Istio, you'll notice the cni-node pods in the istio-system namespace in a CrashLoopBackoff error. Manually edit the `istio-cni-node` daemonset to include the following on the `install-cni` container:
```yaml
securityContext:
privileged: true
```
**Result:** Now you should be able to utilize Istio as desired, including sidecar injection and monitoring via Kiali.
content/rancher/v2.x/en/istio/v2.5/setup/enable-istio-in-cluster/_index.md
+5 -2
View File
@@ -6,8 +6,11 @@ aliases:
- /rancher/v2.x/en/istio/setup/enable-istio-in-cluster
---
>**Prerequisite:** Only a user with the `cluster-admin` [Kubernetes default role](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#user-facing-roles) assigned can configure and install Istio in a Kubernetes cluster.
>**Prerequisites:**
>
>- Only a user with the `cluster-admin` [Kubernetes default role](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#user-facing-roles) assigned can configure and install Istio in a Kubernetes cluster.
>- If you have pod security policies, you will need to install Istio with the CNI enabled. For details, see [this section.](./enable-istio-with-psp)
>- To install Istio on an RKE2 cluster, additional steps are enabled. For details, see [this section.](./rke2)
1. From the **Cluster Explorer**, navigate to available **Charts** in **Apps & Marketplace**
1. Select the Istio chart from the rancher provided charts
content/rancher/v2.x/en/istio/v2.5/setup/enable-istio-in-cluster/enable-istio-with-psp/_index.md
+108
View File
@@ -0,0 +1,108 @@
---
title: Enable Istio with Pod Security Policies
weight: 1
aliases:
- /rancher/v2.x/en/cluster-admin/tools/istio/setup/enable-istio-in-cluster/enable-istio-with-psp
- /rancher/v2.x/en/istio/legacy/setup/enable-istio-in-cluster/enable-istio-with-psp
---
If you have restrictive Pod Security Policies enabled, then Istio may not be able to function correctly, because it needs certain permissions in order to install itself and manage pod infrastructure. In this section, we will configure a cluster with PSPs enabled for an Istio install, and also set up the Istio CNI plugin.
The Istio CNI plugin removes the need for each application pod to have a privileged `NET_ADMIN` container. For further information, see the [Istio CNI Plugin docs](https://istio.io/docs/setup/additional-setup/cni). Please note that the [Istio CNI Plugin is in alpha](https://istio.io/about/feature-stages/).
The steps differ based on the Rancher version.
{{% tabs %}}
{{% tab "v2.5.4+" %}}
> **Prerequisites:**
>
> - The cluster must be an RKE Kubernetes cluster.
> - The cluster must have been created with a default PodSecurityPolicy.
>
> To enable pod security policy support when creating a Kubernetes cluster in the Rancher UI, go to <b>Advanced Options.</b> In the <b>Pod Security Policy Support</b> section, click <b>Enabled.</b> Then select a default pod security policy.
1. [Set the PodSecurityPolicy to unrestricted](#1-set-the-podsecuritypolicy-to-unrestricted)
2. [Enable the CNI](#2-enable-the-cni)
3. [Verify that the CNI is working.](#3-verify-that-the-cni-is-working)
### 1. Set the PodSecurityPolicy to unrestricted
An unrestricted PSP allows Istio to be installed.
Set the PSP to `unrestricted` in the project where is Istio is installed, or the project where you plan to install Istio.
1. From the cluster view of the **Cluster Manager,** select **Projects/Namespaces.**
1. Find the **Project: System** and select the **&#8942; > Edit**.
1. Change the Pod Security Policy option to be unrestricted, then click **Save.**
### 2. Enable the CNI
When installing or upgrading Istio through **Apps & Marketplace,**
1. Click **Components.**
2. Check the box next to **Enabled CNI.**
3. Finish installing or upgrading Istio.
The CNI can also be enabled by editing the `values.yaml`:
```
istio_cni.enabled: true
```
Istio should install successfully with the CNI enabled in the cluster.
### 3. Verify that the CNI is working
Verify that the CNI is working by deploying a [sample application](https://istio.io/latest/docs/examples/bookinfo/) or deploying one of your own applications.
{{% /tab %}}
{{% tab "v2.5.0-v2.5.3" %}}
> **Prerequisites:**
>
> - The cluster must be an RKE Kubernetes cluster.
> - The cluster must have been created with a default PodSecurityPolicy.
>
> To enable pod security policy support when creating a Kubernetes cluster in the Rancher UI, go to <b>Advanced Options.</b> In the <b>Pod Security Policy Support</b> section, click <b>Enabled.</b> Then select a default pod security policy.
1. [Configure the System Project Policy to allow Istio install.](#1-configure-the-system-project-policy-to-allow-istio-install)
2. [Install the CNI plugin in the System project.](#2-install-the-cni-plugin-in-the-system-project)
3. [Install Istio.](#3-install-istio)
### 1. Configure the System Project Policy to allow Istio install
1. From the cluster view of the **Cluster Manager,** select **Projects/Namespaces.**
1. Find the **Project: System** and select the **&#8942; > Edit**.
1. Change the Pod Security Policy option to be unrestricted, then click Save.
### 2. Install the CNI Plugin in the System Project
1. From the main menu of the **Dashboard**, select **Projects/Namespaces**.
1. Select the **Project: System** project.
1. Choose **Tools > Catalogs** in the navigation bar.
1. Add a catalog with the following:
1. Name: istio-cni
1. Catalog URL: https://github.com/istio/cni
1. Branch: The branch that matches your current release, for example: `release-1.4`.
1. From the main menu select **Apps**
1. Click Launch and select istio-cni
1. Update the namespace to be "kube-system"
1. In the answers section, click "Edit as YAML" and paste in the following, then click launch:
```
---
logLevel: "info"
excludeNamespaces:
- "istio-system"
- "kube-system"
```
### 3. Install Istio
Follow the [primary instructions]({{<baseurl>}}/rancher/v2.x/en/cluster-admin/tools/istio/setup/enable-istio-in-cluster/), adding a custom answer: `istio_cni.enabled: true`.
After Istio has finished installing, the Apps page in System Projects should show both istio and `istio-cni` applications deployed successfully. Sidecar injection will now be functional.
{{% /tab %}}
{{% /tabs %}}
content/rancher/v2.x/en/logging/v2.5/_index.md
+13 -5
View File
@@ -82,9 +82,13 @@ According to the [Banzai Cloud documentation,](https://banzaicloud.com/docs/one-
> You can define `outputs` (destinations where you want to send your log messages, for example, Elasticsearch, or and Amazon S3 bucket), and `flows` that use filters and selectors to route log messages to the appropriate outputs. You can also define cluster-wide outputs and flows, for example, to use a centralized output that namespaced users cannot modify.
### Examples
# Examples
**Cluster Output to ElasticSearch:** Let's say you wanted to send all logs in your cluster to an `elasticsearch` cluster. First, we create a cluster output.
Once logging is installed, you can use these examples to help craft your own logging pipeline.
### Cluster Output to ElasticSearch
Let's say you wanted to send all logs in your cluster to an `elasticsearch` cluster. First, we create a cluster output.
```yaml
apiVersion: logging.banzaicloud.io/v1beta1
@@ -116,7 +120,9 @@ spec:
We should now see our configured index with logs in it.
**Output to Splunk:** What if we have an application team who only wants logs from a specific namespaces sent to a `splunk` server? For this case, we can use namespaced outputs and flows.
### Output to Splunk
What if we have an application team who only wants logs from a specific namespaces sent to a `splunk` server? For this case, we can use namespaced outputs and flows.
Before we start, let's set up that team's application: `coolapp`.
@@ -176,7 +182,9 @@ spec:
- "devteam-splunk"
```
**Unsupported Ouput:** For the final example, we create an output to write logs to a destination that is not supported out of the box (e.g. syslog):
### Unsupported Output
For the final example, we create an output to write logs to a destination that is not supported out of the box (e.g. syslog):
```yaml
apiVersion: v1
@@ -266,7 +274,7 @@ spec:
Let's break down what is happening here. First, we create a deployment of a container that has the additional `syslog` plugin and accepts logs forwarded from another `fluentd`. Next we create an output configured as a forwarder to our deployment. The deployment `fluentd` will then forward all logs to the configured `syslog` destination.
> **Note on syslog** The example provides an overview on using unsupported plugins, but `syslog` support is coming in a future Rancher v2.x release.
> **Note on syslog** Official `syslog` support is coming in Rancher v2.5.4. However, this example still provides an overview on using unsupported plugins.
# Working with Taints and Tolerations
content/rancher/v2.x/en/monitoring-alerting/v2.0.x-v2.4.x/notifiers/_index.md
+7 -7
View File
@@ -63,7 +63,7 @@ Set up a notifier so that you can begin configuring and sending alerts.
| URL | From Slack, create a webhook. For instructions, see the [Slack Documentation](https://get.slack.help/hc/en-us/articles/115005265063-Incoming-WebHooks-for-Slack). Then enter the Slack webhook URL. |
| Default Channel | Enter the name of the channel that you want to send alert notifications in the following format: `#<channelname>`. Both public and private channels are supported. |
| Proxy URL | Proxy for the Slack webhook. |
| Send Resolved Alerts | _Available as of v2.3.0_ When enabled, you will receive resolved alerts, such as an alert about high CPU usage after the CPU has returned to normal levels. |
| Send Resolved Alerts | _Available as of v2.3.0_ Whether to send a follow-up notification if an alert has been resolved (e.g. [Resolved] High CPU Usage) |
**Validation:** Click **Test**. If the test is successful, the Slack channel you're configuring for the notifier outputs **Slack setting validated.**
@@ -73,7 +73,7 @@ Set up a notifier so that you can begin configuring and sending alerts.
|----------|----------------------|
| Name | Enter a **Name** for the notifier. |
| Default Recipient Address | Enter the email address that you want to receive the notification. |
| Send Resolved Alerts | _Available as of v2.3.0_ When enabled, you will receive resolved alerts, such as an alert about high CPU usage after the CPU has returned to normal levels. |
| Send Resolved Alerts | _Available as of v2.3.0_ Whether to send a follow-up notification if an alert has been resolved (e.g. [Resolved] High CPU Usage) |
SMTP Server Configuration:
@@ -95,7 +95,7 @@ SMTP Server Configuration:
| Name | Enter a **Name** for the notifier. |
| Default Integration Key | From PagerDuty, create a Prometheus integration. For instructions, see the [PagerDuty Documentation](https://www.pagerduty.com/docs/guides/prometheus-integration-guide/). Then enter the integration key.
| Service Key | The same as the integration key. For instructions on creating a Prometheus integration, see the [PagerDuty Documentation](https://www.pagerduty.com/docs/guides/prometheus-integration-guide/). Then enter the integration key. |
| Send Resolved Alerts | _Available as of v2.3.0_ When enabled, you will receive resolved alerts, such as an alert about high CPU usage after the CPU has returned to normal levels. |
| Send Resolved Alerts | _Available as of v2.3.0_ Whether to send a follow-up notification if an alert has been resolved (e.g. [Resolved] High CPU Usage) |
**Validation:** Click **Test**. If the test is successful, your PagerDuty endpoint outputs **PagerDuty setting validated.**
@@ -106,7 +106,7 @@ SMTP Server Configuration:
| Name | Enter a **Name** for the notifier. |
| URL | Using the app of your choice, create a webhook URL. |
| Proxy URL | Proxy for the webhook. |
| Send Resolved Alerts | _Available as of v2.3.0_ When enabled, you will receive resolved alerts, such as an alert about high CPU usage after the CPU has returned to normal levels. |
| Send Resolved Alerts | _Available as of v2.3.0_ Whether to send a follow-up notification if an alert has been resolved (e.g. [Resolved] High CPU Usage) |
**Validation:** Click **Test**. If the test is successful, the URL you're configuring as a notifier outputs **Webhook setting validated.**
@@ -123,7 +123,7 @@ _Available as of v2.2.0_
| Recipient Type | Party, tag, or user. |
| Default Recipient | The default recipient ID should correspond to the recipient type. It should be the party ID, tag ID or user account that you want to receive the notification. You could get contact information from [Contacts page](https://work.weixin.qq.com/wework_admin/frame#contacts). |
| Proxy URL | If you are using a proxy, enter the proxy URL. |
| Send Resolved Alerts | _Available as of v2.3.0_ When enabled, you will receive resolved alerts, such as an alert about high CPU usage after the CPU has returned to normal levels. |
| Send Resolved Alerts | _Available as of v2.3.0_ Whether to send a follow-up notification if an alert has been resolved (e.g. [Resolved] High CPU Usage) |
**Validation:** Click **Test.** If the test is successful, you should receive an alert message.
@@ -137,7 +137,7 @@ _Available as of v2.4.6_
| Webhook URL | Enter the DingTalk webhook URL. For help setting up the webhook, refer to the [DingTalk documentation.](https://www.alibabacloud.com/help/doc-detail/52872.htm) |
| Secret | Optional: Enter a secret for the DingTalk webhook. |
| Proxy URL | Optional: Enter a proxy for the DingTalk webhook. |
| Send Resolved Alerts | When enabled, you will receive resolved alerts, such as an alert about high CPU usage after the CPU has returned to normal levels. |
| Send Resolved Alerts | Whether to send a follow-up notification if an alert has been resolved (e.g. [Resolved] High CPU Usage) |
**Validation:** Click **Test.** If the test is successful, the DingTalk notifier output is **DingTalk setting validated.**
@@ -150,7 +150,7 @@ _Available as of v2.4.6_
| Name | Enter a **Name** for the notifier. |
| Webhook URL | Enter the Microsoft Teams webhook URL. For help setting up the webhook, refer to the [Teams Documentation.](https://docs.microsoft.com/en-us/microsoftteams/platform/webhooks-and-connectors/how-to/add-incoming-webhook) |
| Proxy URL | Optional: Enter a proxy for the Teams webhook. |
| Send Resolved Alerts | When enabled, you will receive resolved alerts, such as an alert about high CPU usage after the CPU has returned to normal levels. |
| Send Resolved Alerts | Whether to send a follow-up notification if an alert has been resolved (e.g. [Resolved] High CPU Usage) |
**Validation:** Click **Test.** If the test is successful, the Teams notifier output is **MicrosoftTeams setting validated.**
content/rancher/v2.x/en/monitoring-alerting/v2.5/_index.md
+9 -5
View File
@@ -38,7 +38,7 @@ For more information about upgrading the Monitoring app in Rancher 2.5, please r
- [Grafana UI](#grafana-ui)
- [Prometheus UI](#prometheus-ui)
- [Viewing the Prometheus Targets](#viewing-the-prometheus-targets)
- [Viewing the Prometheus Rules](#viewing-the-prometheus-rules)
- [Viewing the PrometheusRules](#viewing-the-prometheus-rules)
- [Viewing Active Alerts in Alertmanager](#viewing-active-alerts-in-alertmanager)
- [Uninstall Monitoring](#uninstall-monitoring)
- [Setting Resource Limits and Requests](#setting-resource-limits-and-requests)
@@ -129,12 +129,14 @@ To see the Prometheus Targets, install `rancher-monitoring`. Then go to the **Cl
<figcaption>Targets in the Prometheus UI</figcaption>
![Prometheus Targets UI]({{<baseurl>}}/img/rancher/prometheus-targets-ui.png)
### Viewing the Prometheus Rules
### Viewing the PrometheusRules
To see the Prometheus Rules, install `rancher-monitoring`. Then go to the **Cluster Explorer.** In the top left corner, click **Cluster Explorer > Monitoring.** Then click **Prometheus Rules.**
To see the PrometheusRules, install `rancher-monitoring`. Then go to the **Cluster Explorer.** In the top left corner, click **Cluster Explorer > Monitoring.** Then click **Prometheus Rules.**
<figcaption>Rules in the Prometheus UI</figcaption>
![Prometheus Rules UI]({{<baseurl>}}/img/rancher/prometheus-rules-ui.png)
![PrometheusRules UI]({{<baseurl>}}/img/rancher/prometheus-rules-ui.png)
For more information on PrometheusRules in Rancher, see [this page.](./configuration/prometheusrules)
### Viewing Active Alerts in Alertmanager
@@ -144,10 +146,12 @@ The Alertmanager handles alerts sent by client applications such as the Promethe
In the Alertmanager UI, you can view your alerts and the current Alertmanager configuration.
To see the Prometheus Rules, install `rancher-monitoring`. Then go to the **Cluster Explorer.** In the top left corner, click **Cluster Explorer > Monitoring.** Then click **Alertmanager.**
To see the PrometheusRules, install `rancher-monitoring`. Then go to the **Cluster Explorer.** In the top left corner, click **Cluster Explorer > Monitoring.** Then click **Alertmanager.**
**Result:** The Alertmanager UI opens in a new tab. For help with configuration, refer to the [official Alertmanager documentation.](https://prometheus.io/docs/alerting/latest/alertmanager/)
For more information on configuring Alertmanager in Rancher, see [this page.](./configuration/alertmanager)
<figcaption>The Alertmanager UI</figcaption>
![Alertmanager UI]({{<baseurl>}}/img/rancher/alertmanager-ui.png)
content/rancher/v2.x/en/monitoring-alerting/v2.5/configuration/_index.md
+12 -50
View File
@@ -13,8 +13,8 @@ For information on configuring custom scrape targets and rules for Prometheus, p
- [Configuring Targets with ServiceMonitors and PodMonitors](#configuring-targets-with-servicemonitors-and-podmonitors)
- [ServiceMonitors](#servicemonitors)
- [PodMonitors](#podmonitors)
- [PrometheusRules](#prometheusrules)
- [Alertmanager Config](#alertmanager-config)
- [PrometheusRules](#prometheusrules)
- [Alertmanager Config](#alertmanager-config)
- [Trusted CA for Notifiers](#trusted-ca-for-notifiers)
- [Additional Scrape Configurations](#additional-scrape-configurations)
- [Examples](#examples)
@@ -45,40 +45,15 @@ For more information about how ServiceMonitors work, refer to the [Prometheus Op
This CRD declaratively specifies how group of pods should be monitored. Any Pods in your cluster that match the labels located within the PodMonitor `selector` field will be monitored based on the `podMetricsEndpoints` specified on the PodMonitor. For more information on what fields can be specified, please look at the [spec](https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/api.md#podmonitorspec) provided by Prometheus Operator.
### PrometheusRules
# PrometheusRules
This CRD defines a group of Prometheus alerting and/or recording rules.
To add a group of alerting / recording rules, you should create a PrometheusRule CR the defines a RuleGroup with your desired rules, each specifying:
For information on configuring PrometheusRules, refer to [this page.](./prometheusrules)
- The name of the new alert / record
- A PromQL expression for the new alert / record
- Labels that should be attached to the alert / record that identify it (e.g. cluster name or severity)
- Annotations that encode any additional important pieces of information that need to be displayed on the notification for an alert (e.g. summary, description, message, runbook URL, etc.). This field is not required for recording rules.
# Alertmanager Config
For more information on what fields can be specified, please look at the [Prometheus Operator spec.](https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/api.md#prometheusrulespec)
### Alertmanager Config
The [Alertmanager Config](https://prometheus.io/docs/alerting/latest/configuration/#configuration-file) Secret contains the configuration of an Alertmanager instance that sends out notifications based on alerts it receives from Prometheus.
By default, Rancher Monitoring deploys a single Alertmanager onto a cluster that uses a default Alertmanager Config Secret. As part of the chart deployment options, you can opt to increase the number of replicas of the Alertmanager deployed onto your cluster that can all be managed using the same underlying Alertmanager Config Secret.
This Secret should be updated or modified any time you want to:
- Add in new notifiers or receivers
- Change the alerts that should be sent to specific notifiers or receivers
- Change the group of alerts that are sent out
> By default, you can either choose to supply an existing Alertmanager Config Secret (i.e. any Secret in the `cattle-monitoring-system` namespace) or allow Rancher Monitoring to deploy a default Alertmanager Config Secret onto your cluster. By default, the Alertmanager Config Secret created by Rancher will never be modified / deleted on an upgrade / uninstall of the `rancher-monitoring` chart to prevent users from losing or overwriting their alerting configuration when executing operations on the chart.
For more information on what fields can be specified in this secret, please look at the [Prometheus Alertmanager docs](https://prometheus.io/docs/alerting/latest/alertmanager/)
The full spec for the Alertmanager configuration file and what it takes in can be found [here.](https://prometheus.io/docs/alerting/latest/configuration/#configuration-file)
The notification integrations are configured with the `receiver`, which is documented [here.](https://prometheus.io/docs/alerting/latest/configuration/#receiver)
For more information, refer to the [official Prometheus documentation about configuring routes.](https://www.prometheus.io/docs/alerting/latest/configuration/#route)
For information on configuring the Alertmanager, refer to [this page.](./alertmanager)
# Trusted CA for Notifiers
@@ -110,25 +85,12 @@ An example PodMonitor can be found [here.](https://github.com/prometheus-operato
### PrometheusRule
Prometheus rule files are held in PrometheusRule custom resources. Use the label selector field ruleSelector in the Prometheus object to define the rule files that you want to be mounted into Prometheus. An example PrometheusRule is on [this page.](https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/user-guides/alerting.md)
For users who are familiar with Prometheus, a PrometheusRule contains the alerting and recording rules that you would normally place in a [Prometheus rule file](https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/).
For a more fine-grained application of PrometheusRules within your cluster, the ruleSelector field on a Prometheus resource allows you to select which PrometheusRules should be loaded onto Prometheus based on the labels attached to the PrometheusRules resources.
An example PrometheusRule is on [this page.](https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/user-guides/alerting.md)
### Alertmanager Config
To set up notifications via Slack, the following Alertmanager Config YAML should be placed into the `alertmanager.yaml` key of the Alertmanager Config Secret, where the `api_url` should be updated to use your Webhook URL from Slack:
```yaml
route:
group_by: ['job']
group_wait: 30s
group_interval: 5m
repeat_interval: 3h
receiver: 'slack-notifications'
receivers:
- name: 'slack-notifications'
slack_configs:
- send_resolved: true
text: '{{ template "slack.rancher.text" . }}'
api_url: <user-provided slack webhook url here>
templates:
- /etc/alertmanager/config/*.tmpl
```
For an example configuration, refer to [this section.](./alertmanager/#example-alertmanager-config)
content/rancher/v2.x/en/monitoring-alerting/v2.5/configuration/alertmanager/_index.md
+235
View File
@@ -0,0 +1,235 @@
---
title: Alertmanager
weight: 1
---
The [Alertmanager Config](https://prometheus.io/docs/alerting/latest/configuration/#configuration-file) Secret contains the configuration of an Alertmanager instance that sends out notifications based on alerts it receives from Prometheus.
- [Overview](#overview)
- [Connecting Routes and PrometheusRules](#connecting-routes-and-prometheusrules)
- [Creating Receivers in the Rancher UI](#creating-receivers-in-the-rancher-ui)
- [Receiver Configuration](#receiver-configuration)
- [Slack](#slack)
- [Email](#email)
- [PagerDuty](#pagerduty)
- [Opsgenie](#opsgenie)
- [Webhook](#webhook)
- [Custom](#custom)
- [Route Configuration](#route-configuration)
- [Receiver](#receiver)
- [Grouping](#grouping)
- [Matching](#matching)
- [Example Alertmanager YAML](#example-alertmanager-yaml)
- [Example Route Config for CIS Scan Alerts](#example-route-config-for-cis-scan-alerts)
# Overview
By default, Rancher Monitoring deploys a single Alertmanager onto a cluster that uses a default Alertmanager Config Secret. As part of the chart deployment options, you can opt to increase the number of replicas of the Alertmanager deployed onto your cluster that can all be managed using the same underlying Alertmanager Config Secret.
This Secret should be updated or modified any time you want to:
- Add in new notifiers or receivers
- Change the alerts that should be sent to specific notifiers or receivers
- Change the group of alerts that are sent out
> By default, you can either choose to supply an existing Alertmanager Config Secret (i.e. any Secret in the `cattle-monitoring-system` namespace) or allow Rancher Monitoring to deploy a default Alertmanager Config Secret onto your cluster. By default, the Alertmanager Config Secret created by Rancher will never be modified / deleted on an upgrade / uninstall of the `rancher-monitoring` chart to prevent users from losing or overwriting their alerting configuration when executing operations on the chart.
For more information on what fields can be specified in this secret, please look at the [Prometheus Alertmanager docs.](https://prometheus.io/docs/alerting/latest/alertmanager/)
The full spec for the Alertmanager configuration file and what it takes in can be found [here.](https://prometheus.io/docs/alerting/latest/configuration/#configuration-file)
For more information, refer to the [official Prometheus documentation about configuring routes.](https://www.prometheus.io/docs/alerting/latest/configuration/#route)
### Connecting Routes and PrometheusRules
When you define a Rule (which is declared within a RuleGroup in a PrometheusRule resource), the [spec of the Rule itself](https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/api.md#rule) contains labels that are used by Prometheus to figure out which Route should receive this Alert. For example, an Alert with the label `team: front-end` will be sent to all Routes that match on that label.
# Creating Receivers in the Rancher UI
_Available as of v2.5.4_
> **Prerequisites:**
>
>- The monitoring application needs to be installed.
>- If you configured monitoring with an existing Alertmanager Secret, it must have a format that is supported by Rancher's UI. Otherwise you will only be able to make changes based on modifying the Alertmanager Secret directly. Note: We are continuing to make enhancements to what kinds of Alertmanager Configurations we can support using the Routes and Receivers UI, so please [file an issue](https://github.com/rancher/rancher/issues/new) if you have a request for a feature enhancement.
To create notification receivers in the Rancher UI,
1. Click **Cluster Explorer > Monitoring** and click **Receiver.**
2. Enter a name for the receiver.
3. Configure one or more providers for the receiver. For help filling out the forms, refer to the configuration options below.
4. Click **Create.**
**Result:** Alerts can be configured to send notifications to the receiver(s).
# Receiver Configuration
The notification integrations are configured with the `receiver`, which is explained in the [Prometheus documentation.](https://prometheus.io/docs/alerting/latest/configuration/#receiver)
Rancher v2.5.4 introduced the capability to configure receivers by filling out forms in the Rancher UI.
{{% tabs %}}
{{% tab "Rancher v2.5.4+" %}}
The following types of receivers can be configured in the Rancher UI:
- <a href="#slack">Slack</Slack>
- <a href="#email">Email</Slack>
- <a href="#pagerduty">PagerDuty</Slack>
- <a href="#opsgenie">Opsgenie</Slack>
- <a href="#webhook">Webhook</Slack>
- <a href="#custom">Custom</Slack>
The custom receiver option can be used to configure any receiver in YAML that cannot be configured by filling out the other forms in the Rancher UI.
### Slack
| Field | Type | Description |
|------|--------------|------|
| URL | String | Enter your Slack webhook URL. For instructions to create a Slack webhook, see the [Slack documentation.](https://get.slack.help/hc/en-us/articles/115005265063-Incoming-WebHooks-for-Slack) |
| Default Channel | String | Enter the name of the channel that you want to send alert notifications in the following format: `#<channelname>`. |
| Proxy URL | String | Proxy for the webhook notifications. |
| Enable Send Resolved Alerts | Bool | Whether to send a follow-up notification if an alert has been resolved (e.g. [Resolved] High CPU Usage). |
### Email
| Field | Type | Description |
|------|--------------|------|
| Default Recipient Address | String | The email address that will receive notifications. |
| Enable Send Resolved Alerts | Bool | Whether to send a follow-up notification if an alert has been resolved (e.g. [Resolved] High CPU Usage). |
SMTP options:
| Field | Type | Description |
|------|--------------|------|
| Sender | String | Enter an email address available on your SMTP mail server that you want to send the notification from. |
| Host | String | Enter the IP address or hostname for your SMTP server. Example: `smtp.email.com`. |
| Use TLS | Bool | Use TLS for encryption. |
| Username | String | Enter a username to authenticate with the SMTP server. |
| Password | String | Enter a password to authenticate with the SMTP server. |
### PagerDuty
| Field | Type | Description |
|------|------|-------|
| Integration Type | String | `Events API v2` or `Prometheus`. |
| Default Integration Key | String | For instructions to get an integration key, see the [PagerDuty documentation.](https://www.pagerduty.com/docs/guides/prometheus-integration-guide/) |
| Proxy URL | String | Proxy for the PagerDuty notifications. |
| Enable Send Resolved Alerts | Bool | Whether to send a follow-up notification if an alert has been resolved (e.g. [Resolved] High CPU Usage). |
### Opsgenie
| Field | Description |
|------|-------------|
| API Key | For instructions to get an API key, refer to the [Opsgenie documentation.](https://docs.opsgenie.com/docs/api-key-management) |
| Proxy URL | Proxy for the Opsgenie notifications. |
| Enable Send Resolved Alerts | Whether to send a follow-up notification if an alert has been resolved (e.g. [Resolved] High CPU Usage). |
Opsgenie Responders:
| Field | Type | Description |
|-------|------|--------|
| Type | String | Schedule, Team, User, or Escalation. For more information on alert responders, refer to the [Opsgenie documentation.](https://docs.opsgenie.com/docs/alert-recipients-and-teams) |
| Send To | String | Id, Name, or Username of the Opsgenie recipient. |
### Webhook
| Field | Description |
|-------|--------------|
| URL | Webhook URL for the app of your choice. |
| Proxy URL | Proxy for the webhook notification. |
| Enable Send Resolved Alerts | Whether to send a follow-up notification if an alert has been resolved (e.g. [Resolved] High CPU Usage). |
### Custom
The YAML provided here will be directly appended to your receiver within the Alertmanager Config Secret.
{{% /tab %}}
{{% tab "Rancher v2.5.0-2.5.3" %}}
The Alertmanager must be configured in YAML, as shown in this [example.](#example-alertmanager-config)
{{% /tab %}}
{{% /tabs %}}
# Route Configuration
{{% tabs %}}
{{% tab "Rancher v2.5.4+" %}}
### Receiver
The route needs to refer to a [receiver](#receiver-configuration) that has already been configured.
### Grouping
| Field | Default | Description |
|-------|--------------|---------|
| Group By | N/a | The labels by which incoming alerts are grouped together. For example, `[ group_by: '[' <labelname>, ... ']' ]` Multiple alerts coming in for labels such as `cluster=A` and `alertname=LatencyHigh` can be batched into a single group. To aggregate by all possible labels, use the special value `'...'` as the sole label name, for example: `group_by: ['...']` Grouping by `...` effectively disables aggregation entirely, passing through all alerts as-is. This is unlikely to be what you want, unless you have a very low alert volume or your upstream notification system performs its own grouping. |
| Group Wait | 30s | How long to wait to buffer alerts of the same group before sending initially. |
| Group Interval | 5m | How long to wait before sending an alert that has been added to a group of alerts for which an initial notification has already been sent. |
| Repeat Interval | 4h | How long to wait before re-sending a given alert that has already been sent. |
### Matching
The **Match** field refers to a set of equality matchers used to identify which alerts to send to a given Route based on labels defined on that alert. When you add key-value pairs to the Rancher UI, they correspond to the YAML in this format:
```yaml
match:
[ <labelname>: <labelvalue>, ... ]
```
The **Match Regex** field refers to a set of regex-matchers used to identify which alerts to send to a given Route based on labels defined on that alert. When you add key-value pairs in the Rancher UI, they correspond to the YAML in this format:
```yaml
match_re:
[ <labelname>: <regex>, ... ]
```
{{% /tab %}}
{{% tab "Rancher v2.5.0-2.5.3" %}}
The Alertmanager must be configured in YAML, as shown in this [example.](#example-alertmanager-config)
{{% /tab %}}
{{% /tabs %}}
# Example Alertmanager Config
To set up notifications via Slack, the following Alertmanager Config YAML can be placed into the `alertmanager.yaml` key of the Alertmanager Config Secret, where the `api_url` should be updated to use your Webhook URL from Slack:
```yaml
route:
group_by: ['job']
group_wait: 30s
group_interval: 5m
repeat_interval: 3h
receiver: 'slack-notifications'
receivers:
- name: 'slack-notifications'
slack_configs:
- send_resolved: true
text: '{{ template "slack.rancher.text" . }}'
api_url: <user-provided slack webhook url here>
templates:
- /etc/alertmanager/config/*.tmpl
```
# Example Route Config for CIS Scan Alerts
While configuring the routes for `rancher-cis-benchmark` alerts, you can specify the matching using the key-value pair `job: rancher-cis-scan`.
For example, the following example route configuration could be used with a Slack receiver named `test-cis`:
```yaml
spec:
receiver: test-cis
group_by:
# - string
group_wait: 30s
group_interval: 30s
repeat_interval: 30s
match:
job: rancher-cis-scan
# key: string
match_re:
{}
# key: string
```
For more information on enabling alerting for `rancher-cis-benchmark`, see [this section.]({{<baseurl>}}/rancher/v2.x/en/cis-scans/v2.5/#enabling-alerting-for-rancher-cis-benchmark)
content/rancher/v2.x/en/monitoring-alerting/v2.5/configuration/expression/_index.md
+432
View File
@@ -0,0 +1,432 @@
---
title: Prometheus Expressions
weight: 4
aliases:
- /rancher/v2.x/en/project-admin/tools/monitoring/expression
- /rancher/v2.x/en/cluster-admin/tools/monitoring/expression
- /rancher/v2.x/en/monitoring-alerting/legacy/monitoring/cluster-monitoring/expression
---
The PromQL expressions in this doc can be used to configure alerts.
For more information about querying Prometheus, refer to the official [Prometheus documentation.](https://prometheus.io/docs/prometheus/latest/querying/basics/)
<!-- TOC -->
- [Cluster Metrics](#cluster-metrics)
- [Cluster CPU Utilization](#cluster-cpu-utilization)
- [Cluster Load Average](#cluster-load-average)
- [Cluster Memory Utilization](#cluster-memory-utilization)
- [Cluster Disk Utilization](#cluster-disk-utilization)
- [Cluster Disk I/O](#cluster-disk-i-o)
- [Cluster Network Packets](#cluster-network-packets)
- [Cluster Network I/O](#cluster-network-i-o)
- [Node Metrics](#node-metrics)
- [Node CPU Utilization](#node-cpu-utilization)
- [Node Load Average](#node-load-average)
- [Node Memory Utilization](#node-memory-utilization)
- [Node Disk Utilization](#node-disk-utilization)
- [Node Disk I/O](#node-disk-i-o)
- [Node Network Packets](#node-network-packets)
- [Node Network I/O](#node-network-i-o)
- [Etcd Metrics](#etcd-metrics)
- [Etcd Has a Leader](#etcd-has-a-leader)
- [Number of Times the Leader Changes](#number-of-times-the-leader-changes)
- [Number of Failed Proposals](#number-of-failed-proposals)
- [GRPC Client Traffic](#grpc-client-traffic)
- [Peer Traffic](#peer-traffic)
- [DB Size](#db-size)
- [Active Streams](#active-streams)
- [Raft Proposals](#raft-proposals)
- [RPC Rate](#rpc-rate)
- [Disk Operations](#disk-operations)
- [Disk Sync Duration](#disk-sync-duration)
- [Kubernetes Components Metrics](#kubernetes-components-metrics)
- [API Server Request Latency](#api-server-request-latency)
- [API Server Request Rate](#api-server-request-rate)
- [Scheduling Failed Pods](#scheduling-failed-pods)
- [Controller Manager Queue Depth](#controller-manager-queue-depth)
- [Scheduler E2E Scheduling Latency](#scheduler-e2e-scheduling-latency)
- [Scheduler Preemption Attempts](#scheduler-preemption-attempts)
- [Ingress Controller Connections](#ingress-controller-connections)
- [Ingress Controller Request Process Time](#ingress-controller-request-process-time)
- [Rancher Logging Metrics](#rancher-logging-metrics)
- [Fluentd Buffer Queue Rate](#fluentd-buffer-queue-rate)
- [Fluentd Input Rate](#fluentd-input-rate)
- [Fluentd Output Errors Rate](#fluentd-output-errors-rate)
- [Fluentd Output Rate](#fluentd-output-rate)
- [Workload Metrics](#workload-metrics)
- [Workload CPU Utilization](#workload-cpu-utilization)
- [Workload Memory Utilization](#workload-memory-utilization)
- [Workload Network Packets](#workload-network-packets)
- [Workload Network I/O](#workload-network-i-o)
- [Workload Disk I/O](#workload-disk-i-o)
- [Pod Metrics](#pod-metrics)
- [Pod CPU Utilization](#pod-cpu-utilization)
- [Pod Memory Utilization](#pod-memory-utilization)
- [Pod Network Packets](#pod-network-packets)
- [Pod Network I/O](#pod-network-i-o)
- [Pod Disk I/O](#pod-disk-i-o)
- [Container Metrics](#container-metrics)
- [Container CPU Utilization](#container-cpu-utilization)
- [Container Memory Utilization](#container-memory-utilization)
- [Container Disk I/O](#container-disk-i-o)
<!-- /TOC -->
# Cluster Metrics
### Cluster CPU Utilization
| Catalog | Expression |
| --- | --- |
| Detail | `1 - (avg(irate(node_cpu_seconds_total{mode="idle"}[5m])) by (instance))` |
| Summary | `1 - (avg(irate(node_cpu_seconds_total{mode="idle"}[5m])))` |
### Cluster Load Average
| Catalog | Expression |
| --- | --- |
| Detail | <table><tr><td>load1</td><td>`sum(node_load1) by (instance) / count(node_cpu_seconds_total{mode="system"}) by (instance)`</td></tr><tr><td>load5</td><td>`sum(node_load5) by (instance) / count(node_cpu_seconds_total{mode="system"}) by (instance)`</td></tr><tr><td>load15</td><td>`sum(node_load15) by (instance) / count(node_cpu_seconds_total{mode="system"}) by (instance)`</td></tr></table> |
| Summary | <table><tr><td>load1</td><td>`sum(node_load1) by (instance) / count(node_cpu_seconds_total{mode="system"})`</td></tr><tr><td>load5</td><td>`sum(node_load5) by (instance) / count(node_cpu_seconds_total{mode="system"})`</td></tr><tr><td>load15</td><td>`sum(node_load15) by (instance) / count(node_cpu_seconds_total{mode="system"})`</td></tr></table> |
### Cluster Memory Utilization
| Catalog | Expression |
| --- | --- |
| Detail | `1 - sum(node_memory_MemAvailable_bytes) by (instance) / sum(node_memory_MemTotal_bytes) by (instance)` |
| Summary | `1 - sum(node_memory_MemAvailable_bytes) / sum(node_memory_MemTotal_bytes)` |
### Cluster Disk Utilization
| Catalog | Expression |
| --- | --- |
| Detail | `(sum(node_filesystem_size_bytes{device!="rootfs"}) by (instance) - sum(node_filesystem_free_bytes{device!="rootfs"}) by (instance)) / sum(node_filesystem_size_bytes{device!="rootfs"}) by (instance)` |
| Summary | `(sum(node_filesystem_size_bytes{device!="rootfs"}) - sum(node_filesystem_free_bytes{device!="rootfs"})) / sum(node_filesystem_size_bytes{device!="rootfs"})` |
### Cluster Disk I/O
| Catalog | Expression |
| --- | --- |
| Detail | <table><tr><td>read</td><td>`sum(rate(node_disk_read_bytes_total[5m])) by (instance)`</td></tr><tr><td>written</td><td>`sum(rate(node_disk_written_bytes_total[5m])) by (instance)`</td></tr></table> |
| Summary | <table><tr><td>read</td><td>`sum(rate(node_disk_read_bytes_total[5m]))`</td></tr><tr><td>written</td><td>`sum(rate(node_disk_written_bytes_total[5m]))`</td></tr></table> |
### Cluster Network Packets
| Catalog | Expression |
| --- | --- |
| Detail | <table><tr><td>receive-dropped</td><td><code>sum(rate(node_network_receive_drop_total{device!~"lo &#124; veth.* &#124; docker.* &#124; flannel.* &#124; cali.* &#124; cbr.*"}[5m])) by (instance)</code></td></tr><tr><td>receive-errs</td><td><code>sum(rate(node_network_receive_errs_total{device!~"lo &#124; veth.* &#124; docker.* &#124; flannel.* &#124; cali.* &#124; cbr.*"}[5m])) by (instance)</code></td></tr><tr><td>receive-packets</td><td><code>sum(rate(node_network_receive_packets_total{device!~"lo &#124; veth.* &#124; docker.* &#124; flannel.* &#124; cali.* &#124; cbr.*"}[5m])) by (instance)</code></td></tr><tr><td>transmit-dropped</td><td><code>sum(rate(node_network_transmit_drop_total{device!~"lo &#124; veth.* &#124; docker.* &#124; flannel.* &#124; cali.* &#124; cbr.*"}[5m])) by (instance)</code></td></tr><tr><td>transmit-errs</td><td><code>sum(rate(node_network_transmit_errs_total{device!~"lo &#124; veth.* &#124; docker.* &#124; flannel.* &#124; cali.* &#124; cbr.*"}[5m])) by (instance)</code></td></tr><tr><td>transmit-packets</td><td><code>sum(rate(node_network_transmit_packets_total{device!~"lo &#124; veth.* &#124; docker.* &#124; flannel.* &#124; cali.* &#124; cbr.*"}[5m])) by (instance)</code></td></tr></table> |
| Summary | <table><tr><td>receive-dropped</td><td><code>sum(rate(node_network_receive_drop_total{device!~"lo &#124; veth.* &#124; docker.* &#124; flannel.* &#124; cali.* &#124; cbr.*"}[5m]))</code></td></tr><tr><td>receive-errs</td><td><code>sum(rate(node_network_receive_errs_total{device!~"lo &#124; veth.* &#124; docker.* &#124; flannel.* &#124; cali.* &#124; cbr.*"}[5m]))</code></td></tr><tr><td>receive-packets</td><td><code>sum(rate(node_network_receive_packets_total{device!~"lo &#124; veth.* &#124; docker.* &#124; flannel.* &#124; cali.* &#124; cbr.*"}[5m]))</code></td></tr><tr><td>transmit-dropped</td><td><code>sum(rate(node_network_transmit_drop_total{device!~"lo &#124; veth.* &#124; docker.* &#124; flannel.* &#124; cali.* &#124; cbr.*"}[5m]))</code></td></tr><tr><td>transmit-errs</td><td><code>sum(rate(node_network_transmit_errs_total{device!~"lo &#124; veth.* &#124; docker.* &#124; flannel.* &#124; cali.* &#124; cbr.*"}[5m]))</code></td></tr><tr><td>transmit-packets</td><td><code>sum(rate(node_network_transmit_packets_total{device!~"lo &#124; veth.* &#124; docker.* &#124; flannel.* &#124; cali.* &#124; cbr.*"}[5m]))</code></td></tr></table> |
### Cluster Network I/O
| Catalog | Expression |
| --- | --- |
| Detail | <table><tr><td>receive</td><td><code>sum(rate(node_network_receive_bytes_total{device!~"lo &#124; veth.* &#124; docker.* &#124; flannel.* &#124; cali.* &#124; cbr.*"}[5m])) by (instance)</code></td></tr><tr><td>transmit</td><td><code>sum(rate(node_network_transmit_bytes_total{device!~"lo &#124; veth.* &#124; docker.* &#124; flannel.* &#124; cali.* &#124; cbr.*"}[5m])) by (instance)</code></td></tr></table> |
| Summary | <table><tr><td>receive</td><td><code>sum(rate(node_network_receive_bytes_total{device!~"lo &#124; veth.* &#124; docker.* &#124; flannel.* &#124; cali.* &#124; cbr.*"}[5m]))</code></td></tr><tr><td>transmit</td><td><code>sum(rate(node_network_transmit_bytes_total{device!~"lo &#124; veth.* &#124; docker.* &#124; flannel.* &#124; cali.* &#124; cbr.*"}[5m]))</code></td></tr></table> |
# Node Metrics
### Node CPU Utilization
| Catalog | Expression |
| --- | --- |
| Detail | `avg(irate(node_cpu_seconds_total{mode!="idle", instance=~"$instance"}[5m])) by (mode)` |
| Summary | `1 - (avg(irate(node_cpu_seconds_total{mode="idle", instance=~"$instance"}[5m])))` |
### Node Load Average
| Catalog | Expression |
| --- | --- |
| Detail | <table><tr><td>load1</td><td>`sum(node_load1{instance=~"$instance"}) / count(node_cpu_seconds_total{mode="system",instance=~"$instance"})`</td></tr><tr><td>load5</td><td>`sum(node_load5{instance=~"$instance"}) / count(node_cpu_seconds_total{mode="system",instance=~"$instance"})`</td></tr><tr><td>load15</td><td>`sum(node_load15{instance=~"$instance"}) / count(node_cpu_seconds_total{mode="system",instance=~"$instance"})`</td></tr></table> |
| Summary | <table><tr><td>load1</td><td>`sum(node_load1{instance=~"$instance"}) / count(node_cpu_seconds_total{mode="system",instance=~"$instance"})`</td></tr><tr><td>load5</td><td>`sum(node_load5{instance=~"$instance"}) / count(node_cpu_seconds_total{mode="system",instance=~"$instance"})`</td></tr><tr><td>load15</td><td>`sum(node_load15{instance=~"$instance"}) / count(node_cpu_seconds_total{mode="system",instance=~"$instance"})`</td></tr></table> |
### Node Memory Utilization
| Catalog | Expression |
| --- | --- |
| Detail | `1 - sum(node_memory_MemAvailable_bytes{instance=~"$instance"}) / sum(node_memory_MemTotal_bytes{instance=~"$instance"})` |
| Summary | `1 - sum(node_memory_MemAvailable_bytes{instance=~"$instance"}) / sum(node_memory_MemTotal_bytes{instance=~"$instance"}) ` |
### Node Disk Utilization
| Catalog | Expression |
| --- | --- |
| Detail | `(sum(node_filesystem_size_bytes{device!="rootfs",instance=~"$instance"}) by (device) - sum(node_filesystem_free_bytes{device!="rootfs",instance=~"$instance"}) by (device)) / sum(node_filesystem_size_bytes{device!="rootfs",instance=~"$instance"}) by (device)` |
| Summary | `(sum(node_filesystem_size_bytes{device!="rootfs",instance=~"$instance"}) - sum(node_filesystem_free_bytes{device!="rootfs",instance=~"$instance"})) / sum(node_filesystem_size_bytes{device!="rootfs",instance=~"$instance"})` |
### Node Disk I/O
| Catalog | Expression |
| --- | --- |
| Detail | <table><tr><td>read</td><td>`sum(rate(node_disk_read_bytes_total{instance=~"$instance"}[5m]))`</td></tr><tr><td>written</td><td>`sum(rate(node_disk_written_bytes_total{instance=~"$instance"}[5m]))`</td></tr></table> |
| Summary | <table><tr><td>read</td><td>`sum(rate(node_disk_read_bytes_total{instance=~"$instance"}[5m]))`</td></tr><tr><td>written</td><td>`sum(rate(node_disk_written_bytes_total{instance=~"$instance"}[5m]))`</td></tr></table> |
### Node Network Packets
| Catalog | Expression |
| --- | --- |
| Detail | <table><tr><td>receive-dropped</td><td><code>sum(rate(node_network_receive_drop_total{device!~"lo &#124; veth.* &#124; docker.* &#124; flannel.* &#124; cali.* &#124; cbr.*",instance=~"$instance"}[5m])) by (device)</code></td></tr><tr><td>receive-errs</td><td><code>sum(rate(node_network_receive_errs_total{device!~"lo &#124; veth.* &#124; docker.* &#124; flannel.* &#124; cali.* &#124; cbr.*",instance=~"$instance"}[5m])) by (device)</code></td></tr><tr><td>receive-packets</td><td><code>sum(rate(node_network_receive_packets_total{device!~"lo &#124; veth.* &#124; docker.* &#124; flannel.* &#124; cali.* &#124; cbr.*",instance=~"$instance"}[5m])) by (device)</code></td></tr><tr><td>transmit-dropped</td><td><code>sum(rate(node_network_transmit_drop_total{device!~"lo &#124; veth.* &#124; docker.* &#124; flannel.* &#124; cali.* &#124; cbr.*",instance=~"$instance"}[5m])) by (device)</code></td></tr><tr><td>transmit-errs</td><td><code>sum(rate(node_network_transmit_errs_total{device!~"lo &#124; veth.* &#124; docker.* &#124; flannel.* &#124; cali.* &#124; cbr.*",instance=~"$instance"}[5m])) by (device)</code></td></tr><tr><td>transmit-packets</td><td><code>sum(rate(node_network_transmit_packets_total{device!~"lo &#124; veth.* &#124; docker.* &#124; flannel.* &#124; cali.* &#124; cbr.*",instance=~"$instance"}[5m])) by (device)</code></td></tr></table> |
| Summary | <table><tr><td>receive-dropped</td><td><code>sum(rate(node_network_receive_drop_total{device!~"lo &#124; veth.* &#124; docker.* &#124; flannel.* &#124; cali.* &#124; cbr.*",instance=~"$instance"}[5m]))</code></td></tr><tr><td>receive-errs</td><td><code>sum(rate(node_network_receive_errs_total{device!~"lo &#124; veth.* &#124; docker.* &#124; flannel.* &#124; cali.* &#124; cbr.*",instance=~"$instance"}[5m]))</code></td></tr><tr><td>receive-packets</td><td><code>sum(rate(node_network_receive_packets_total{device!~"lo &#124; veth.* &#124; docker.* &#124; flannel.* &#124; cali.* &#124; cbr.*",instance=~"$instance"}[5m]))</code></td></tr><tr><td>transmit-dropped</td><td><code>sum(rate(node_network_transmit_drop_total{device!~"lo &#124; veth.* &#124; docker.* &#124; flannel.* &#124; cali.* &#124; cbr.*",instance=~"$instance"}[5m]))</code></td></tr><tr><td>transmit-errs</td><td><code>sum(rate(node_network_transmit_errs_total{device!~"lo &#124; veth.* &#124; docker.* &#124; flannel.* &#124; cali.* &#124; cbr.*",instance=~"$instance"}[5m]))</code></td></tr><tr><td>transmit-packets</td><td><code>sum(rate(node_network_transmit_packets_total{device!~"lo &#124; veth.* &#124; docker.* &#124; flannel.* &#124; cali.* &#124; cbr.*",instance=~"$instance"}[5m]))</code></td></tr></table> |
### Node Network I/O
| Catalog | Expression |
| --- | --- |
| Detail | <table><tr><td>receive</td><td><code>sum(rate(node_network_receive_bytes_total{device!~"lo &#124; veth.* &#124; docker.* &#124; flannel.* &#124; cali.* &#124; cbr.*",instance=~"$instance"}[5m])) by (device)</code></td></tr><tr><td>transmit</td><td><code>sum(rate(node_network_transmit_bytes_total{device!~"lo &#124; veth.* &#124; docker.* &#124; flannel.* &#124; cali.* &#124; cbr.*",instance=~"$instance"}[5m])) by (device)</code></td></tr></table> |
| Summary | <table><tr><td>receive</td><td><code>sum(rate(node_network_receive_bytes_total{device!~"lo &#124; veth.* &#124; docker.* &#124; flannel.* &#124; cali.* &#124; cbr.*",instance=~"$instance"}[5m]))</code></td></tr><tr><td>transmit</td><td><code>sum(rate(node_network_transmit_bytes_total{device!~"lo &#124; veth.* &#124; docker.* &#124; flannel.* &#124; cali.* &#124; cbr.*",instance=~"$instance"}[5m]))</code></td></tr></table> |
# Etcd Metrics
### Etcd Has a Leader
`max(etcd_server_has_leader)`
### Number of Times the Leader Changes
`max(etcd_server_leader_changes_seen_total)`
### Number of Failed Proposals
`sum(etcd_server_proposals_failed_total)`
### GRPC Client Traffic
| Catalog | Expression |
| --- | --- |
| Detail | <table><tr><td>in</td><td>`sum(rate(etcd_network_client_grpc_received_bytes_total[5m])) by (instance)`</td></tr><tr><td>out</td><td>`sum(rate(etcd_network_client_grpc_sent_bytes_total[5m])) by (instance)`</td></tr></table> |
| Summary | <table><tr><td>in</td><td>`sum(rate(etcd_network_client_grpc_received_bytes_total[5m]))`</td></tr><tr><td>out</td><td>`sum(rate(etcd_network_client_grpc_sent_bytes_total[5m]))`</td></tr></table> |
### Peer Traffic
| Catalog | Expression |
| --- | --- |
| Detail | <table><tr><td>in</td><td>`sum(rate(etcd_network_peer_received_bytes_total[5m])) by (instance)`</td></tr><tr><td>out</td><td>`sum(rate(etcd_network_peer_sent_bytes_total[5m])) by (instance)`</td></tr></table> |
| Summary | <table><tr><td>in</td><td>`sum(rate(etcd_network_peer_received_bytes_total[5m]))`</td></tr><tr><td>out</td><td>`sum(rate(etcd_network_peer_sent_bytes_total[5m]))`</td></tr></table> |
### DB Size
| Catalog | Expression |
| --- | --- |
| Detail | `sum(etcd_debugging_mvcc_db_total_size_in_bytes) by (instance)` |
| Summary | `sum(etcd_debugging_mvcc_db_total_size_in_bytes)` |
### Active Streams
| Catalog | Expression |
| --- | --- |
| Detail | <table><tr><td>lease-watch</td><td>`sum(grpc_server_started_total{grpc_service="etcdserverpb.Lease",grpc_type="bidi_stream"}) by (instance) - sum(grpc_server_handled_total{grpc_service="etcdserverpb.Lease",grpc_type="bidi_stream"}) by (instance)`</td></tr><tr><td>watch</td><td>`sum(grpc_server_started_total{grpc_service="etcdserverpb.Watch",grpc_type="bidi_stream"}) by (instance) - sum(grpc_server_handled_total{grpc_service="etcdserverpb.Watch",grpc_type="bidi_stream"}) by (instance)`</td></tr></table> |
| Summary | <table><tr><td>lease-watch</td><td>`sum(grpc_server_started_total{grpc_service="etcdserverpb.Lease",grpc_type="bidi_stream"}) - sum(grpc_server_handled_total{grpc_service="etcdserverpb.Lease",grpc_type="bidi_stream"})`</td></tr><tr><td>watch</td><td>`sum(grpc_server_started_total{grpc_service="etcdserverpb.Watch",grpc_type="bidi_stream"}) - sum(grpc_server_handled_total{grpc_service="etcdserverpb.Watch",grpc_type="bidi_stream"})`</td></tr></table> |
### Raft Proposals
| Catalog | Expression |
| --- | --- |
| Detail | <table><tr><td>applied</td><td>`sum(increase(etcd_server_proposals_applied_total[5m])) by (instance)`</td></tr><tr><td>committed</td><td>`sum(increase(etcd_server_proposals_committed_total[5m])) by (instance)`</td></tr><tr><td>pending</td><td>`sum(increase(etcd_server_proposals_pending[5m])) by (instance)`</td></tr><tr><td>failed</td><td>`sum(increase(etcd_server_proposals_failed_total[5m])) by (instance)`</td></tr></table> |
| Summary | <table><tr><td>applied</td><td>`sum(increase(etcd_server_proposals_applied_total[5m]))`</td></tr><tr><td>committed</td><td>`sum(increase(etcd_server_proposals_committed_total[5m]))`</td></tr><tr><td>pending</td><td>`sum(increase(etcd_server_proposals_pending[5m]))`</td></tr><tr><td>failed</td><td>`sum(increase(etcd_server_proposals_failed_total[5m]))`</td></tr></table> |
### RPC Rate
| Catalog | Expression |
| --- | --- |
| Detail | <table><tr><td>total</td><td>`sum(rate(grpc_server_started_total{grpc_type="unary"}[5m])) by (instance)`</td></tr><tr><td>fail</td><td>`sum(rate(grpc_server_handled_total{grpc_type="unary",grpc_code!="OK"}[5m])) by (instance)`</td></tr></table> |
| Summary | <table><tr><td>total</td><td>`sum(rate(grpc_server_started_total{grpc_type="unary"}[5m]))`</td></tr><tr><td>fail</td><td>`sum(rate(grpc_server_handled_total{grpc_type="unary",grpc_code!="OK"}[5m]))`</td></tr></table> |
### Disk Operations
| Catalog | Expression |
| --- | --- |
| Detail | <table><tr><td>commit-called-by-backend</td><td>`sum(rate(etcd_disk_backend_commit_duration_seconds_sum[1m])) by (instance)`</td></tr><tr><td>fsync-called-by-wal</td><td>`sum(rate(etcd_disk_wal_fsync_duration_seconds_sum[1m])) by (instance)`</td></tr></table> |
| Summary | <table><tr><td>commit-called-by-backend</td><td>`sum(rate(etcd_disk_backend_commit_duration_seconds_sum[1m]))`</td></tr><tr><td>fsync-called-by-wal</td><td>`sum(rate(etcd_disk_wal_fsync_duration_seconds_sum[1m]))`</td></tr></table> |
### Disk Sync Duration
| Catalog | Expression |
| --- | --- |
| Detail | <table><tr><td>wal</td><td>`histogram_quantile(0.99, sum(rate(etcd_disk_wal_fsync_duration_seconds_bucket[5m])) by (instance, le))`</td></tr><tr><td>db</td><td>`histogram_quantile(0.99, sum(rate(etcd_disk_backend_commit_duration_seconds_bucket[5m])) by (instance, le))`</td></tr></table> |
| Summary | <table><tr><td>wal</td><td>`sum(histogram_quantile(0.99, sum(rate(etcd_disk_wal_fsync_duration_seconds_bucket[5m])) by (instance, le)))`</td></tr><tr><td>db</td><td>`sum(histogram_quantile(0.99, sum(rate(etcd_disk_backend_commit_duration_seconds_bucket[5m])) by (instance, le)))`</td></tr></table> |
# Kubernetes Components Metrics
### API Server Request Latency
| Catalog | Expression |
| --- | --- |
| Detail | `avg(apiserver_request_latencies_sum / apiserver_request_latencies_count) by (instance, verb) /1e+06` |
| Summary | `avg(apiserver_request_latencies_sum / apiserver_request_latencies_count) by (instance) /1e+06` |
### API Server Request Rate
| Catalog | Expression |
| --- | --- |
| Detail | `sum(rate(apiserver_request_count[5m])) by (instance, code)` |
| Summary | `sum(rate(apiserver_request_count[5m])) by (instance)` |
### Scheduling Failed Pods
| Catalog | Expression |
| --- | --- |
| Detail | `sum(kube_pod_status_scheduled{condition="false"})` |
| Summary | `sum(kube_pod_status_scheduled{condition="false"})` |
### Controller Manager Queue Depth
| Catalog | Expression |
| --- | --- |
| Detail | <table><tr><td>volumes</td><td>`sum(volumes_depth) by instance`</td></tr><tr><td>deployment</td><td>`sum(deployment_depth) by instance`</td></tr><tr><td>replicaset</td><td>`sum(replicaset_depth) by instance`</td></tr><tr><td>service</td><td>`sum(service_depth) by instance`</td></tr><tr><td>serviceaccount</td><td>`sum(serviceaccount_depth) by instance`</td></tr><tr><td>endpoint</td><td>`sum(endpoint_depth) by instance`</td></tr><tr><td>daemonset</td><td>`sum(daemonset_depth) by instance`</td></tr><tr><td>statefulset</td><td>`sum(statefulset_depth) by instance`</td></tr><tr><td>replicationmanager</td><td>`sum(replicationmanager_depth) by instance`</td></tr></table> |
| Summary | <table><tr><td>volumes</td><td>`sum(volumes_depth)`</td></tr><tr><td>deployment</td><td>`sum(deployment_depth)`</td></tr><tr><td>replicaset</td><td>`sum(replicaset_depth)`</td></tr><tr><td>service</td><td>`sum(service_depth)`</td></tr><tr><td>serviceaccount</td><td>`sum(serviceaccount_depth)`</td></tr><tr><td>endpoint</td><td>`sum(endpoint_depth)`</td></tr><tr><td>daemonset</td><td>`sum(daemonset_depth)`</td></tr><tr><td>statefulset</td><td>`sum(statefulset_depth)`</td></tr><tr><td>replicationmanager</td><td>`sum(replicationmanager_depth)`</td></tr></table> |
### Scheduler E2E Scheduling Latency
| Catalog | Expression |
| --- | --- |
| Detail | `histogram_quantile(0.99, sum(scheduler_e2e_scheduling_latency_microseconds_bucket) by (le, instance)) / 1e+06` |
| Summary | `sum(histogram_quantile(0.99, sum(scheduler_e2e_scheduling_latency_microseconds_bucket) by (le, instance)) / 1e+06)` |
### Scheduler Preemption Attempts
| Catalog | Expression |
| --- | --- |
| Detail | `sum(rate(scheduler_total_preemption_attempts[5m])) by (instance)` |
| Summary | `sum(rate(scheduler_total_preemption_attempts[5m]))` |
### Ingress Controller Connections
| Catalog | Expression |
| --- | --- |
| Detail | <table><tr><td>reading</td><td>`sum(nginx_ingress_controller_nginx_process_connections{state="reading"}) by (instance)`</td></tr><tr><td>waiting</td><td>`sum(nginx_ingress_controller_nginx_process_connections{state="waiting"}) by (instance)`</td></tr><tr><td>writing</td><td>`sum(nginx_ingress_controller_nginx_process_connections{state="writing"}) by (instance)`</td></tr><tr><td>accepted</td><td>`sum(ceil(increase(nginx_ingress_controller_nginx_process_connections_total{state="accepted"}[5m]))) by (instance)`</td></tr><tr><td>active</td><td>`sum(ceil(increase(nginx_ingress_controller_nginx_process_connections_total{state="active"}[5m]))) by (instance)`</td></tr><tr><td>handled</td><td>`sum(ceil(increase(nginx_ingress_controller_nginx_process_connections_total{state="handled"}[5m]))) by (instance)`</td></tr></table> |
| Summary | <table><tr><td>reading</td><td>`sum(nginx_ingress_controller_nginx_process_connections{state="reading"})`</td></tr><tr><td>waiting</td><td>`sum(nginx_ingress_controller_nginx_process_connections{state="waiting"})`</td></tr><tr><td>writing</td><td>`sum(nginx_ingress_controller_nginx_process_connections{state="writing"})`</td></tr><tr><td>accepted</td><td>`sum(ceil(increase(nginx_ingress_controller_nginx_process_connections_total{state="accepted"}[5m])))`</td></tr><tr><td>active</td><td>`sum(ceil(increase(nginx_ingress_controller_nginx_process_connections_total{state="active"}[5m])))`</td></tr><tr><td>handled</td><td>`sum(ceil(increase(nginx_ingress_controller_nginx_process_connections_total{state="handled"}[5m])))`</td></tr></table> |
### Ingress Controller Request Process Time
| Catalog | Expression |
| --- | --- |
| Detail | `topk(10, histogram_quantile(0.95,sum by (le, host, path)(rate(nginx_ingress_controller_request_duration_seconds_bucket{host!="_"}[5m]))))` |
| Summary | `topk(10, histogram_quantile(0.95,sum by (le, host)(rate(nginx_ingress_controller_request_duration_seconds_bucket{host!="_"}[5m]))))` |
# Rancher Logging Metrics
### Fluentd Buffer Queue Rate
| Catalog | Expression |
| --- | --- |
| Detail | `sum(rate(fluentd_output_status_buffer_queue_length[5m])) by (instance)` |
| Summary | `sum(rate(fluentd_output_status_buffer_queue_length[5m]))` |
### Fluentd Input Rate
| Catalog | Expression |
| --- | --- |
| Detail | `sum(rate(fluentd_input_status_num_records_total[5m])) by (instance)` |
| Summary | `sum(rate(fluentd_input_status_num_records_total[5m]))` |
### Fluentd Output Errors Rate
| Catalog | Expression |
| --- | --- |
| Detail | `sum(rate(fluentd_output_status_num_errors[5m])) by (type)` |
| Summary | `sum(rate(fluentd_output_status_num_errors[5m]))` |
### Fluentd Output Rate
| Catalog | Expression |
| --- | --- |
| Detail | `sum(rate(fluentd_output_status_num_records_total[5m])) by (instance)` |
| Summary | `sum(rate(fluentd_output_status_num_records_total[5m]))` |
# Workload Metrics
### Workload CPU Utilization
| Catalog | Expression |
| --- | --- |
| Detail | <table><tr><td>cfs throttled seconds</td><td>`sum(rate(container_cpu_cfs_throttled_seconds_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`</td></tr><tr><td>user seconds</td><td>`sum(rate(container_cpu_user_seconds_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`</td></tr><tr><td>system seconds</td><td>`sum(rate(container_cpu_system_seconds_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`</td></tr><tr><td>usage seconds</td><td>`sum(rate(container_cpu_usage_seconds_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`</td></tr></table> |
| Summary | <table><tr><td>cfs throttled seconds</td><td>`sum(rate(container_cpu_cfs_throttled_seconds_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`</td></tr><tr><td>user seconds</td><td>`sum(rate(container_cpu_user_seconds_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`</td></tr><tr><td>system seconds</td><td>`sum(rate(container_cpu_system_seconds_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`</td></tr><tr><td>usage seconds</td><td>`sum(rate(container_cpu_usage_seconds_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`</td></tr></table> |
### Workload Memory Utilization
| Catalog | Expression |
| --- | --- |
| Detail | `sum(container_memory_working_set_bytes{namespace="$namespace",pod_name=~"$podName", container_name!=""}) by (pod_name)` |
| Summary | `sum(container_memory_working_set_bytes{namespace="$namespace",pod_name=~"$podName", container_name!=""})` |
### Workload Network Packets
| Catalog | Expression |
| --- | --- |
| Detail | <table><tr><td>receive-packets</td><td>`sum(rate(container_network_receive_packets_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`</td></tr><tr><td>receive-dropped</td><td>`sum(rate(container_network_receive_packets_dropped_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`</td></tr><tr><td>receive-errors</td><td>`sum(rate(container_network_receive_errors_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`</td></tr><tr><td>transmit-packets</td><td>`sum(rate(container_network_transmit_packets_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`</td></tr><tr><td>transmit-dropped</td><td>`sum(rate(container_network_transmit_packets_dropped_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`</td></tr><tr><td>transmit-errors</td><td>`sum(rate(container_network_transmit_errors_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`</td></tr></table> |
| Summary | <table><tr><td>receive-packets</td><td>`sum(rate(container_network_receive_packets_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`</td></tr><tr><td>receive-dropped</td><td>`sum(rate(container_network_receive_packets_dropped_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`</td></tr><tr><td>receive-errors</td><td>`sum(rate(container_network_receive_errors_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`</td></tr><tr><td>transmit-packets</td><td>`sum(rate(container_network_transmit_packets_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`</td></tr><tr><td>transmit-dropped</td><td>`sum(rate(container_network_transmit_packets_dropped_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`</td></tr><tr><td>transmit-errors</td><td>`sum(rate(container_network_transmit_errors_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`</td></tr></table> |
### Workload Network I/O
| Catalog | Expression |
| --- | --- |
| Detail | <table><tr><td>receive</td><td>`sum(rate(container_network_receive_bytes_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`</td></tr><tr><td>transmit</td><td>`sum(rate(container_network_transmit_bytes_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`</td></tr></table> |
| Summary | <table><tr><td>receive</td><td>`sum(rate(container_network_receive_bytes_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`</td></tr><tr><td>transmit</td><td>`sum(rate(container_network_transmit_bytes_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`</td></tr></table> |
### Workload Disk I/O
| Catalog | Expression |
| --- | --- |
| Detail | <table><tr><td>read</td><td>`sum(rate(container_fs_reads_bytes_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`</td></tr><tr><td>write</td><td>`sum(rate(container_fs_writes_bytes_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`</td></tr></table> |
| Summary | <table><tr><td>read</td><td>`sum(rate(container_fs_reads_bytes_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`</td></tr><tr><td>write</td><td>`sum(rate(container_fs_writes_bytes_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`</td></tr></table> |
# Pod Metrics
### Pod CPU Utilization
| Catalog | Expression |
| --- | --- |
| Detail | <table><tr><td>cfs throttled seconds</td><td>`sum(rate(container_cpu_cfs_throttled_seconds_total{container_name!="POD",namespace="$namespace",pod_name="$podName", container_name!=""}[5m])) by (container_name)`</td></tr><tr><td>usage seconds</td><td>`sum(rate(container_cpu_usage_seconds_total{container_name!="POD",namespace="$namespace",pod_name="$podName", container_name!=""}[5m])) by (container_name)`</td></tr><tr><td>system seconds</td><td>`sum(rate(container_cpu_system_seconds_total{container_name!="POD",namespace="$namespace",pod_name="$podName", container_name!=""}[5m])) by (container_name)`</td></tr><tr><td>user seconds</td><td>`sum(rate(container_cpu_user_seconds_total{container_name!="POD",namespace="$namespace",pod_name="$podName", container_name!=""}[5m])) by (container_name)`</td></tr></table> |
| Summary | <table><tr><td>cfs throttled seconds</td><td>`sum(rate(container_cpu_cfs_throttled_seconds_total{container_name!="POD",namespace="$namespace",pod_name="$podName", container_name!=""}[5m]))`</td></tr><tr><td>usage seconds</td><td>`sum(rate(container_cpu_usage_seconds_total{container_name!="POD",namespace="$namespace",pod_name="$podName", container_name!=""}[5m]))`</td></tr><tr><td>system seconds</td><td>`sum(rate(container_cpu_system_seconds_total{container_name!="POD",namespace="$namespace",pod_name="$podName", container_name!=""}[5m]))`</td></tr><tr><td>user seconds</td><td>`sum(rate(container_cpu_user_seconds_total{container_name!="POD",namespace="$namespace",pod_name="$podName", container_name!=""}[5m]))`</td></tr></table> |
### Pod Memory Utilization
| Catalog | Expression |
| --- | --- |
| Detail | `sum(container_memory_working_set_bytes{container_name!="POD",namespace="$namespace",pod_name="$podName",container_name!=""}) by (container_name)` |
| Summary | `sum(container_memory_working_set_bytes{container_name!="POD",namespace="$namespace",pod_name="$podName",container_name!=""})` |
### Pod Network Packets
| Catalog | Expression |
| --- | --- |
| Detail | <table><tr><td>receive-packets</td><td>`sum(rate(container_network_receive_packets_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr><tr><td>receive-dropped</td><td>`sum(rate(container_network_receive_packets_dropped_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr><tr><td>receive-errors</td><td>`sum(rate(container_network_receive_errors_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr><tr><td>transmit-packets</td><td>`sum(rate(container_network_transmit_packets_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr><tr><td>transmit-dropped</td><td>`sum(rate(container_network_transmit_packets_dropped_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr><tr><td>transmit-errors</td><td>`sum(rate(container_network_transmit_errors_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr></table> |
| Summary | <table><tr><td>receive-packets</td><td>`sum(rate(container_network_receive_packets_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr><tr><td>receive-dropped</td><td>`sum(rate(container_network_receive_packets_dropped_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr><tr><td>receive-errors</td><td>`sum(rate(container_network_receive_errors_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr><tr><td>transmit-packets</td><td>`sum(rate(container_network_transmit_packets_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr><tr><td>transmit-dropped</td><td>`sum(rate(container_network_transmit_packets_dropped_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr><tr><td>transmit-errors</td><td>`sum(rate(container_network_transmit_errors_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr></table> |
### Pod Network I/O
| Catalog | Expression |
| --- | --- |
| Detail | <table><tr><td>receive</td><td>`sum(rate(container_network_receive_bytes_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr><tr><td>transmit</td><td>`sum(rate(container_network_transmit_bytes_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr></table> |
| Summary | <table><tr><td>receive</td><td>`sum(rate(container_network_receive_bytes_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr><tr><td>transmit</td><td>`sum(rate(container_network_transmit_bytes_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr></table> |
### Pod Disk I/O
| Catalog | Expression |
| --- | --- |
| Detail | <table><tr><td>read</td><td>`sum(rate(container_fs_reads_bytes_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m])) by (container_name)`</td></tr><tr><td>write</td><td>`sum(rate(container_fs_writes_bytes_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m])) by (container_name)`</td></tr></table> |
| Summary | <table><tr><td>read</td><td>`sum(rate(container_fs_reads_bytes_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr><tr><td>write</td><td>`sum(rate(container_fs_writes_bytes_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr></table> |
# Container Metrics
### Container CPU Utilization
| Catalog | Expression |
| --- | --- |
| cfs throttled seconds | `sum(rate(container_cpu_cfs_throttled_seconds_total{namespace="$namespace",pod_name="$podName",container_name="$containerName"}[5m]))` |
| usage seconds | `sum(rate(container_cpu_usage_seconds_total{namespace="$namespace",pod_name="$podName",container_name="$containerName"}[5m]))` |
| system seconds | `sum(rate(container_cpu_system_seconds_total{namespace="$namespace",pod_name="$podName",container_name="$containerName"}[5m]))` |
| user seconds | `sum(rate(container_cpu_user_seconds_total{namespace="$namespace",pod_name="$podName",container_name="$containerName"}[5m]))` |
### Container Memory Utilization
`sum(container_memory_working_set_bytes{namespace="$namespace",pod_name="$podName",container_name="$containerName"})`
### Container Disk I/O
| Catalog | Expression |
| --- | --- |
| read | `sum(rate(container_fs_reads_bytes_total{namespace="$namespace",pod_name="$podName",container_name="$containerName"}[5m]))` |
| write | `sum(rate(container_fs_writes_bytes_total{namespace="$namespace",pod_name="$podName",container_name="$containerName"}[5m]))` |
content/rancher/v2.x/en/monitoring-alerting/v2.5/configuration/prometheusrules/_index.md
+98
View File
@@ -0,0 +1,98 @@
---
title: PrometheusRules
weight: 2
---
A PrometheusRule defines a group of Prometheus alerting and/or recording rules.
- [About PrometheusRule Custom Resources](#about-prometheusrule-custom-resources)
- [Connecting Routes and PrometheusRules](#connecting-routes-and-prometheusrules)
- [Creating PrometheusRules in the Rancher UI](#creating-prometheusrules-in-the-rancher-ui)
- [Configuration](#configuration)
- [Rule Group](#rule-group)
- [Alerting Rules](#alerting-rules)
- [Recording Rules](#recording-rules)
### About PrometheusRule Custom Resources
Prometheus rule files are held in PrometheusRule custom resources.
A PrometheusRule allows you to define one or more RuleGroups. Each RuleGroup consists of a set of Rule objects that can each represent either an alerting or a recording rule with the following fields:
- The name of the new alert or record
- A PromQL (Prometheus query language) expression for the new alert or record
- Labels that should be attached to the alert or record that identify it (e.g. cluster name or severity)
- Annotations that encode any additional important pieces of information that need to be displayed on the notification for an alert (e.g. summary, description, message, runbook URL, etc.). This field is not required for recording rules.
Alerting rules define alert conditions based on PromQL queries. Recording rules precompute frequently needed or computationally expensive queries at defined intervals.
For more information on what fields can be specified, please look at the [Prometheus Operator spec.](https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/api.md#prometheusrulespec)
Use the label selector field `ruleSelector` in the Prometheus object to define the rule files that you want to be mounted into Prometheus.
For examples, refer to the Prometheus documentation on [recording rules](https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/) and [alerting rules.](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/)
### Connecting Routes and PrometheusRules
When you define a Rule (which is declared within a RuleGroup in a PrometheusRule resource), the [spec of the Rule itself](https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/api.md#rule) contains labels that are used by Prometheus to figure out which Route should receive this Alert. For example, an Alert with the label `team: front-end` will be sent to all Routes that match on that label.
### Creating PrometheusRules in the Rancher UI
_Available as of v2.5.4_
> **Prerequisite:** The monitoring application needs to be installed.
To create rule groups in the Rancher UI,
1. Click **Cluster Explorer > Monitoring** and click **Prometheus Rules.**
1. Click **Create.**
1. Enter a **Group Name.**
1. Configure the rules. In Rancher's UI, we expect a rule group to contain either alert rules or recording rules, but not both. For help filling out the forms, refer to the configuration options below.
1. Click **Create.**
**Result:** Alerts can be configured to send notifications to the receiver(s).
# Configuration
{{% tabs %}}
{{% tab "Rancher v2.5.4" %}}
Rancher v2.5.4 introduced the capability to configure PrometheusRules by filling out forms in the Rancher UI.
### Rule Group
| Field | Description |
|-------|----------------|
| Group Name | The name of the group. Must be unique within a rules file. |
| Override Group Interval | Duration in seconds for how often rules in the group are evaluated. |
### Alerting Rules
[Alerting rules](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/) allow you to define alert conditions based on PromQL (Prometheus expression language) expressions and to send notifications about firing alerts to an external service.
| Field | Description |
|-------|----------------|
| Alert Name | The name of the alert. Must be a valid label value. |
| Wait To Fire For | Duration in seconds. Alerts are considered firing once they have been returned for this long. Alerts which have not yet fired for long enough are considered pending. |
| PromQL Expression | The PromQL expression to evaluate. Prometheus will evaluate the current value of this PromQL expression on every evaluation cycle and all resultant time series will become pending/firing alerts. For more information, refer to the [Prometheus documentation](https://prometheus.io/docs/prometheus/latest/querying/basics/) or our [example PromQL expressions.](../expression) |
| Labels | Labels to add or overwrite for each alert. |
| Severity | When enabled, labels are attached to the alert or record that identify it by the severity level. |
| Severity Label Value | Critical, warning, or none |
| Annotations | Annotations are a set of informational labels that can be used to store longer additional information, such as alert descriptions or runbook links. A [runbook](https://en.wikipedia.org/wiki/Runbook) is a set of documentation about how to handle alerts. The annotation values can be [templated.](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/#templating) |
### Recording Rules
[Recording rules](https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/#recording-rules) allow you to precompute frequently needed or computationally expensive PromQL (Prometheus expression language) expressions and save their result as a new set of time series.
| Field | Description |
|-------|----------------|
| Time Series Name | The name of the time series to output to. Must be a valid metric name. |
| PromQL Expression | The PromQL expression to evaluate. Prometheus will evaluate the current value of this PromQL expression on every evaluation cycle and the result will be recorded as a new set of time series with the metric name as given by 'record'. For more information about expressions, refer to the [Prometheus documentation](https://prometheus.io/docs/prometheus/latest/querying/basics/) or our [example PromQL expressions.](../expression) |
| Labels | Labels to add or overwrite before storing the result. |
{{% /tab %}}
{{% tab "Rancher v2.5.0-v2.5.3" %}}
For Rancher v2.5.0-v2.5.3, PrometheusRules must be configured in YAML. For examples, refer to the Prometheus documentation on [recording rules](https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/) and [alerting rules.](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/)
{{% /tab %}}
{{% /tabs %}}
content/rke/latest/en/config-options/add-ons/_index.md
+6
View File
@@ -49,3 +49,9 @@ _Applies to v0.2.3 and higher_
| kube-dns | - `beta.kubernetes.io/os:NotIn:windows`<br/>- `node-role.kubernetes.io/worker` `Exists` | none | - `NoSchedule:Exists`<br/>- `NoExecute:Exists`<br/>- `CriticalAddonsOnly:Exists` |
| nginx-ingress | - `beta.kubernetes.io/os:NotIn:windows`<br/>- `node-role.kubernetes.io/worker` `Exists` | none | - `NoSchedule:Exists`<br/>- `NoExecute:Exists` |
| metrics-server | - `beta.kubernetes.io/os:NotIn:windows`<br/>- `node-role.kubernetes.io/worker` `Exists` | none | - `NoSchedule:Exists`<br/>- `NoExecute:Exists` |
## Tolerations
_Available as of v1.2.4_
Tolerations can be configured per add-on and apply to Deployment resources. The configured tolerations will replace the existing tolerations so make sure you configure all the tolerations you need. See the specific add-on doc pages for more information.
content/rke/latest/en/config-options/add-ons/dns/_index.md
+83 -28
View File
@@ -34,16 +34,16 @@ If you only want the CoreDNS pod to be deployed on specific nodes, you can set a
```yaml
nodes:
- address: 1.1.1.1
role: [controlplane,worker,etcd]
user: root
labels:
app: dns
- address: 1.1.1.1
role: [controlplane,worker,etcd]
user: root
labels:
app: dns
dns:
provider: coredns
node_selector:
app: dns
provider: coredns
node_selector:
app: dns
```
## Configuring CoreDNS
@@ -56,10 +56,37 @@ When you set `upstreamnameservers`, the `provider` also needs to be set.
```yaml
dns:
provider: coredns
upstreamnameservers:
- 1.1.1.1
- 8.8.4.4
provider: coredns
upstreamnameservers:
- 1.1.1.1
- 8.8.4.4
```
### Tolerations
_Available as of v1.2.4_
The configured tolerations apply to the `coredns` and the `coredns-autoscaler` Deployment.
```
dns:
provider: coredns
tolerations:
- key: "node.kubernetes.io/unreachable"
operator: "Exists"
effect: "NoExecute"
tolerationseconds: 300
- key: "node.kubernetes.io/not-ready"
operator: "Exists"
effect: "NoExecute"
tolerationseconds: 300
```
To check for applied tolerations on the `coredns` and `coredns-autoscaler` Deployment, use the following commands:
```
kubectl -n kube-system get deploy coredns -o jsonpath='{.spec.template.spec.tolerations}'
kubectl -n kube-system get deploy coredns-autoscaler -o jsonpath='{.spec.template.spec.tolerations}'
```
# kube-dns
@@ -76,16 +103,16 @@ If you only want the kube-dns pod to be deployed on specific nodes, you can set
```yaml
nodes:
- address: 1.1.1.1
role: [controlplane,worker,etcd]
user: root
labels:
app: dns
- address: 1.1.1.1
role: [controlplane,worker,etcd]
user: root
labels:
app: dns
dns:
provider: kube-dns
node_selector:
app: dns
provider: kube-dns
node_selector:
app: dns
```
## Configuring kube-dns
@@ -100,10 +127,38 @@ When you set `upstreamnameservers`, the `provider` also needs to be set.
```yaml
dns:
provider: kube-dns
upstreamnameservers:
- 1.1.1.1
- 8.8.4.4
provider: kube-dns
upstreamnameservers:
- 1.1.1.1
- 8.8.4.4
```
### Tolerations
_Available as of v1.2.4_
The configured tolerations apply to the `kube-dns` and the `kube-dns-autoscaler` Deployment.
```
dns:
provider: kube-dns
tolerations:
- key: "node.kubernetes.io/unreachable"
operator: "Exists"
effect: "NoExecute"
tolerationseconds: 300
- key: "node.kubernetes.io/not-ready"
operator: "Exists"
effect: "NoExecute"
tolerationseconds: 300
```
To check for applied tolerations on the `coredns` and `coredns-autoscaler` Deployment, use the following commands:
```
kubectl get deploy kube-dns -n kube-system -o jsonpath='{.spec.template.spec.tolerations}'
kubectl get deploy kube-dns-autoscaler -n kube-system -o jsonpath='{.spec.template.spec.tolerations}'
```
# Disabling deployment of a DNS provider
@@ -114,7 +169,7 @@ You can disable the default DNS provider by specifying `none` to the dns `provi
```yaml
dns:
provider: none
provider: none
```
# NodeLocal DNS
@@ -137,9 +192,9 @@ The `ip_address` parameter is used to configure what link-local IP address will
```yaml
dns:
provider: coredns
nodelocal:
ip_address: "169.254.20.10"
provider: coredns
nodelocal:
ip_address: "169.254.20.10"
```
> **Note:** When enabling NodeLocal DNS on an existing cluster, pods that are currently running will not be modified, the updated `/etc/resolv.conf` configuration will take effect only for pods started after enabling NodeLocal DNS.
content/rke/latest/en/config-options/add-ons/ingress-controllers/_index.md
+50 -14
View File
@@ -18,16 +18,41 @@ If you only wanted ingress controllers to be deployed on specific nodes, you can
```yaml
nodes:
- address: 1.1.1.1
role: [controlplane,worker,etcd]
user: root
labels:
app: ingress
- address: 1.1.1.1
role: [controlplane,worker,etcd]
user: root
labels:
app: ingress
ingress:
provider: nginx
node_selector:
app: ingress
provider: nginx
node_selector:
app: ingress
```
## Tolerations
_Available as of v1.2.4_
The configured tolerations apply to the `default-http-backend` Deployment.
```
ingress:
tolerations:
- key: "node.kubernetes.io/unreachable"
operator: "Exists"
effect: "NoExecute"
tolerationseconds: 300
- key: "node.kubernetes.io/not-ready"
operator: "Exists"
effect: "NoExecute"
tolerationseconds: 300
```
To check for applied tolerations `default-http-backend` Deployment, use the following commands:
```
kubectl -n ingress-nginx get deploy default-http-backend -o jsonpath='{.spec.template.spec.tolerations}'
```
## Disabling the Default Ingress Controller
@@ -44,14 +69,25 @@ For the configuration of NGINX, there are configuration options available in Kub
```yaml
ingress:
provider: nginx
options:
map-hash-bucket-size: "128"
ssl-protocols: SSLv2
extra_args:
enable-ssl-passthrough: ""
provider: nginx
options:
map-hash-bucket-size: "128"
ssl-protocols: SSLv2
extra_args:
enable-ssl-passthrough: ""
```
### Disabling NGINX Ingress Default Backend
As of v0.20.0, you can disable the [default backend service](https://kubernetes.github.io/ingress-nginx/user-guide/default-backend/) for the ingress controller. This is possible because `ingress-nginx` will fall back to a local 404 page, and does not require a backend service. The service can be enabled/disabled with a boolean value.
```yaml
ingress:
default_backend: false
```
> **What happens if the field is omitted?** The value of `default_backend` will default to `true`. This maintains behavior with older versions of `rke`. However, a future version of `rke` will change the default value to `false`.
## Configuring an NGINX Default Certificate
When configuring an ingress object with TLS termination, you must provide it with a certificate used for encryption/decryption. Instead of explicitly defining a certificate each time you configure an ingress, you can set up a custom certificate that's used by default.
content/rke/latest/en/config-options/add-ons/metrics-server/_index.md
+26 -1
View File
@@ -9,6 +9,31 @@ RKE will deploy Metrics Server as a Deployment.
The image used for Metrics Server is under the [`system_images` directive]({{<baseurl>}}/rke/latest/en/config-options/system-images/). For each Kubernetes version, there is a default image associated with the Metrics Server, but these can be overridden by changing the image tag in `system_images`.
## Tolerations
_Available as of v1.2.4_
The configured tolerations apply to the `metrics-server` Deployment.
```
monitoring:
tolerations:
- key: "node.kubernetes.io/unreachable"
operator: "Exists"
effect: "NoExecute"
tolerationseconds: 300
- key: "node.kubernetes.io/not-ready"
operator: "Exists"
effect: "NoExecute"
tolerationseconds: 300
```
To check for applied tolerations on the `metrics-server` Deployment, use the following commands:
```
kubectl -n kube-system get deploy metrics-server -o jsonpath='{.spec.template.spec.tolerations}'
```
## Disabling the Metrics Server
_Available as of v0.2.0_
@@ -17,5 +42,5 @@ You can disable the default controller by specifying `none` to the monitoring `p
```yaml
monitoring:
provider: none
provider: none
```
content/rke/latest/en/config-options/add-ons/network-plugins/_index.md
+68 -19
View File
@@ -17,7 +17,7 @@ By default, the network plug-in is `canal`. If you want to use another network p
```yaml
# Setting the flannel network plug-in
network:
plugin: flannel
plugin: flannel
```
The images used for network plug-ins are under the [`system_images` directive]({{<baseurl>}}/rke/latest/en/config-options/system-images/). For each Kubernetes version, there are default images associated with each network plug-in, but these can be overridden by changing the image tag in `system_images`.
@@ -28,7 +28,7 @@ You can disable deploying a network plug-in by specifying `none` to the network
```yaml
network:
plugin: none
plugin: none
```
# Network Plug-in Options
@@ -39,11 +39,10 @@ Besides the different images that could be used to deploy network plug-ins, cert
```yaml
network:
plugin: canal
options:
canal_iface: eth1
canal_flannel_backend_type: vxlan
canal_flannel_backend_port: "8472"
plugin: canal
options:
canal_iface: eth1
canal_flannel_backend_type: vxlan
```
#### Canal Interface
@@ -51,15 +50,40 @@ network:
By setting the `canal_iface`, you can configure the interface to use for inter-host communication.
The `canal_flannel_backend_type` option allows you to specify the type of [flannel backend](https://github.com/coreos/flannel/blob/master/Documentation/backends.md) to use. By default the `vxlan` backend is used.
## Canal Network Plug-in Tolerations
_Available as of v1.2.4_
The configured tolerations apply to the `calico-kube-controllers` Deployment.
```
network:
plugin: canal
tolerations:
- key: "node.kubernetes.io/unreachable"
operator: "Exists"
effect: "NoExecute"
tolerationseconds: 300
- key: "node.kubernetes.io/not-ready"
operator: "Exists"
effect: "NoExecute"
tolerationseconds: 300
```
To check for applied tolerations on the `calico-kube-controllers` Deployment, use the following command:
```
kubectl -n kube-system get deploy calico-kube-controllers -o jsonpath='{.spec.template.spec.tolerations}'
```
## Flannel Network Plug-in Options
```yaml
network:
plugin: flannel
options:
flannel_iface: eth1
flannel_backend_type: vxlan
flannel_backend_port: "8472"
plugin: flannel
options:
flannel_iface: eth1
flannel_backend_type: vxlan
```
#### Flannel Interface
@@ -71,9 +95,9 @@ The `flannel_backend_type` option allows you to specify the type of [flannel bac
```yaml
network:
plugin: calico
options:
calico_cloud_provider: aws
plugin: calico
options:
calico_cloud_provider: aws
```
#### Calico Cloud Provider
@@ -84,20 +108,45 @@ Calico currently only supports 2 cloud providers, AWS or GCE, which can be set u
- `aws`
- `gce`
## Calico Network Plug-in Tolerations
_Available as of v1.2.4_
The configured tolerations apply to the `calico-kube-controllers` Deployment.
```
network:
plugin: calico
tolerations:
- key: "node.kubernetes.io/unreachable"
operator: "Exists"
effect: "NoExecute"
tolerationseconds: 300
- key: "node.kubernetes.io/not-ready"
operator: "Exists"
effect: "NoExecute"
tolerationseconds: 300
```
To check for applied tolerations on the `calico-kube-controllers` Deployment, use the following command:
```
kubectl -n kube-system get deploy calico-kube-controllers -o jsonpath='{.spec.template.spec.tolerations}'
```
## Weave Network Plug-in Options
```yaml
network:
plugin: weave
weave_network_provider:
password: "Q]SZOQ5wp@n$oijz"
plugin: weave
weave_network_provider:
password: "Q]SZOQ5wp@n$oijz"
```
#### Weave encryption
Weave encryption can be enabled by passing a string password to the network provider config.
## Custom Network Plug-ins
It is possible to add a custom network plug-in by using the [user-defined add-on functionality]({{<baseurl>}}/rke/latest/en/config-options/add-ons/user-defined-add-ons/) of RKE. In the `addons` field, you can add the add-on manifest of a cluster that has the network plugin-that you want, as shown in [this example.]({{<baseurl>}}/rke/latest/en/config-options/add-ons/network-plugins/custom-network-plugin-example)
content/rke/latest/en/os/_index.md
+47
View File
@@ -16,6 +16,7 @@ weight: 5
- [OpenSSH version](#openssh-version)
- [Creating a Docker Group](#creating-a-docker-group)
- [Flatcar Container Linux](#flatcar-container-linux)
- [Software](#software)
- [Ports](#ports)
@@ -162,6 +163,52 @@ By default, Atomic hosts do not come with a Docker group. You can update the own
# chown <user> /var/run/docker.sock
```
### Flatcar Container Linux
When using Flatcar Container Linux nodes, it is required to use the following configuration in the cluster configuration file:
{{% tabs %}}
{{% tab "Canal"%}}
```yaml
rancher_kubernetes_engine_config:
network:
plugin: canal
options:
canal_flex_volume_plugin_dir: /opt/kubernetes/kubelet-plugins/volume/exec/nodeagent~uds
flannel_backend_type: vxlan
services:
kube-controller:
extra_args:
flex-volume-plugin-dir: /opt/kubernetes/kubelet-plugins/volume/exec/
```
{{% /tab %}}
{{% tab "Calico"%}}
```yaml
rancher_kubernetes_engine_config:
network:
plugin: calico
options:
calico_flex_volume_plugin_dir: /opt/kubernetes/kubelet-plugins/volume/exec/nodeagent~uds
flannel_backend_type: vxlan
services:
kube-controller:
extra_args:
flex-volume-plugin-dir: /opt/kubernetes/kubelet-plugins/volume/exec/
```
{{% /tab %}}
{{% /tabs %}}
It is also required to enable the Docker service, you can enable the Docker service using the following command:
```
systemctl enable docker.service
```
## Software
This section describes the requirements for Docker, Kubernetes, and SSH.
layouts/shortcodes/ports-custom-nodes.html
+11 -36
View File
@@ -6,7 +6,7 @@
<th>etcd Plane Nodes</th>
<th>Control Plane Nodes</th>
<th>Worker Plane Nodes</th>
<th>External Load Balancer</th>
<th>External Rancher Load Balancer</th>
<th>Internet</th>
</tr>
</thead>
@@ -46,7 +46,7 @@
<td></td>
</tr>
<tr>
<td colspan="3" style="background-color: #3497DA; color:#ffffff;">4789 UDP <sup>(7)</sup></td>
<td colspan="3" style="background-color: #3497DA; color:#ffffff;">4789 UDP <sup>(6)</sup></td>
<td></td>
</tr>
<tr>
@@ -81,7 +81,7 @@
<td></td>
</tr>
<tr>
<td colspan="3" style="background-color: #3497DA; color:#ffffff;">4789 UDP <sup>(7)</sup></td>
<td colspan="3" style="background-color: #3497DA; color:#ffffff;">4789 UDP <sup>(6)</sup></td>
<td></td>
</tr>
<tr>
@@ -114,7 +114,7 @@
<td></td>
</tr>
<tr>
<td colspan="3" style="background-color: #3497DA; color:#ffffff;">4789 UDP <sup>(7)</sup></td>
<td colspan="3" style="background-color: #3497DA; color:#ffffff;">4789 UDP <sup>(6)</sup></td>
<td></td>
</tr>
<tr>
@@ -130,41 +130,16 @@
<td></td>
</tr>
<tr>
<td rowspan="2">External Load Balancer <sup>(5)</sup></td>
<td style="background-color: #3497DA; color:#ffffff;">80 TCP</td>
<td>Kubernetes API Clients</td>
<td></td>
<td></td>
<td style="background-color: #3497DA; color:#ffffff;">6443 TCP <sup>(5)</sup></td>
<td></td>
<td></td>
<td></td>
</tr>
<tr>
<td style="background-color: #3497DA; color:#ffffff;">443 TCP <sup>(6)</sup></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
</tr>
<tr>
<td rowspan="2">API / UI Clients</td>
<td style="background-color: #3497DA; color:#ffffff;">80 TCP <sup>(3)</sup></td>
<td></td>
<td></td>
<td></td>
<td style="background-color: #3497DA; color:#ffffff;">80 TCP<br></td>
<td></td>
</tr>
<tr>
<td style="background-color: #3497DA; color:#ffffff;">443 TCP <sup>(3)</sup></td>
<td></td>
<td></td>
<td></td>
<td style="background-color: #3497DA; color:#ffffff;">443 TCP</td>
<td></td>
</tr>
<tr>
<td rowspan="3">Workload Clients</td>
<td rowspan="3">Workload Clients or Load Balancer</td>
<td></td>
<td></td>
<td></td>
@@ -175,19 +150,19 @@
<tr>
<td></td>
<td></td>
<td colspan="2" style="background-color: #3497DA; color:#ffffff;">80 TCP (Ingress)</td>
<td></td>
<td style="background-color: #3497DA; color:#ffffff;">80 TCP (Ingress)</td>
<td></td>
</tr>
<tr>
<td></td>
<td></td>
<td colspan="2" style="background-color: #3497DA; color:#ffffff;">443 TCP (Ingress)</td>
<td></td>
<td style="background-color: #3497DA; color:#ffffff;">443 TCP (Ingress)</td>
<td></td>
</tr>
<tr>
<td colspan="7">Notes:<br><br>1. Nodes running standalone server or Rancher HA deployment.<br>2. Required to fetch Rancher chart library.<br>3. Only without external load balancer.<br>4. Local traffic to the node itself (not across nodes).<br>5. Load balancer / proxy that handles tragging to the Rancher UI / API.<br>6. Only if SSL is not terminated at external load balancer.<br>7. Only if using Overlay mode on Windows cluster.</td>
<td colspan="7">Notes:<br><br>1. Nodes running standalone server or Rancher HA deployment.<br>2. Required to fetch Rancher chart library.<br>3. Only without external load balancer in front of Rancher.<br>4. Local traffic to the node itself (not across nodes).<br>5. Only if Authorized Cluster Endpoints are activated.<br>6. Only if using Overlay mode on Windows cluster.</td>
</tr>
</tbody>
</table>
</table>
layouts/shortcodes/ports-iaas-nodes.html
+8 -33
View File
@@ -6,7 +6,7 @@
<th>etcd Plane Nodes</th>
<th>Control Plane Nodes</th>
<th>Worker Plane Nodes</th>
<th>External Load Balancer</th>
<th>External Rancher Load Balancer</th>
<th>Internet</th>
</tr>
</thead>
@@ -121,41 +121,16 @@
<td></td>
</tr>
<tr>
<td rowspan="2">External Load Balancer <sup>(5)</sup></td>
<td style="background-color: #3497DA; color:#ffffff;">80 TCP</td>
<td>Kubernetes API Clients</td>
<td></td>
<td></td>
<td style="background-color: #3497DA; color:#ffffff;">6443 TCP <sup>(5)</sup></td>
<td></td>
<td></td>
<td></td>
</tr>
<tr>
<td style="background-color: #3497DA; color:#ffffff;">443 TCP <sup>(6)</sup></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
</tr>
<tr>
<td rowspan="2">API / UI Clients</td>
<td style="background-color: #3497DA; color:#ffffff;">80 TCP <sup>(3)</sup></td>
<td></td>
<td></td>
<td></td>
<td style="background-color: #3497DA; color:#ffffff;">80 TCP<br></td>
<td></td>
</tr>
<tr>
<td style="background-color: #3497DA; color:#ffffff;">443 TCP <sup>(3)</sup></td>
<td></td>
<td></td>
<td></td>
<td style="background-color: #3497DA; color:#ffffff;">443 TCP</td>
<td></td>
</tr>
<tr>
<td rowspan="3">Workload Clients</td>
<td rowspan="3">Workload Clients or Load Balancer</td>
<td></td>
<td></td>
<td></td>
@@ -166,19 +141,19 @@
<tr>
<td></td>
<td></td>
<td colspan="2" style="background-color: #3497DA; color:#ffffff;">80 TCP (Ingress)</td>
<td></td>
<td style="background-color: #3497DA; color:#ffffff;">80 TCP (Ingress)</td>
<td></td>
</tr>
<tr>
<td></td>
<td></td>
<td colspan="2" style="background-color: #3497DA; color:#ffffff;">443 TCP (Ingress)</td>
<td></td>
<td style="background-color: #3497DA; color:#ffffff;">443 TCP (Ingress)</td>
<td></td>
</tr>
<tr>
<td colspan="7">Notes:<br><br>1. Nodes running standalone server or Rancher HA deployment.<br>2. Required to fetch Rancher chart library.<br>3. Only without external load balancer.<br>4. Local traffic to the node itself (not across nodes).<br>5. Load balancer / proxy that handles tragging to the Rancher UI / API.<br>6. Only if SSL is not terminated at external load balancer.</td>
<td colspan="7">Notes:<br><br>1. Nodes running standalone server or Rancher HA deployment.<br>2. Required to fetch Rancher chart library.<br>3. Only without external load balancer in front of Rancher.<br>4. Local traffic to the node itself (not across nodes).<br>5. Only if Authorized Cluster Endpoints are activated.</td>
</tr>
</tbody>
</table>
</table>
layouts/shortcodes/ports-imported-hosted.html
+17 -14
View File
@@ -4,17 +4,27 @@
<th>From / To</th>
<th>Rancher Nodes</th>
<th>Hosted / Imported Cluster</th>
<th>External Load Balancer</th>
<th>External Rancher Load Balancer</th>
<th>Internet</th>
</tr>
</thead>
<tbody>
<tr>
<td>Rancher Nodes <sup>(1)</sup></td>
<td rowspan="3">Rancher Nodes <sup>(1)</sup></td>
<td></td>
<td style="background-color: #3497DA; color:#ffffff;">Kubernetes API <br>Endpoint Port <sup>(2)</sup></td>
<td></td>
<td style="background-color: #3497DA; color:#ffffff;">git.rancher.io <sup>(3)</sup>:<br>35.160.43.145:32<br>35.167.242.46:32<br>52.33.59.17:32</td>
<td rowspan="3" style="background-color: #3497DA; color:#ffffff;">git.rancher.io <sup>(3)</sup>:<br>35.160.43.145:32<br>35.167.242.46:32<br>52.33.59.17:32</td>
</tr>
<tr>
<td></td>
<td style="background-color: #3497DA; color:#ffffff;">8443 TCP</td>
<td></td>
</tr>
<tr>
<td></td>
<td style="background-color: #3497DA; color:#ffffff;">9443 TCP</td>
<td></td>
</tr>
<tr>
<td>Hosted / Imported Cluster</td>
@@ -24,20 +34,13 @@
<td></td>
</tr>
<tr>
<td>External Load Balancer <sup>(5)</sup></td>
<td style="background-color: #3497DA; color:#ffffff;">80 TCP<br>443 TCP <sup>(6)</sup></td>
<td>Kubernetes API Clients</td>
<td></td>
<td style="background-color: #3497DA; color:#ffffff;">Cluster / Provider Specific <sup>(6)</sup></td>
<td></td>
<td></td>
</tr>
<tr>
<td>API / UI Clients</td>
<td style="background-color: #3497DA; color:#ffffff;">80 TCP <sup>(4)</sup><br>443 TCP <sup>(4)</sup></td>
<td></td>
<td style="background-color: #3497DA; color:#ffffff;">80 TCP<br>443 TCP</td>
<td></td>
</tr>
<tr>
<td>Workload Client</td>
<td></td>
<td style="background-color: #3497DA; color:#ffffff;">Cluster / Provider Specific <sup>(7)</sup></td>
@@ -45,7 +48,7 @@
<td></td>
</tr>
<tr>
<td colspan="5">Notes:<br><br>1. Nodes running standalone server or Rancher HA deployment.<br>2. Only for hosted clusters.<br>3. Required to fetch Rancher chart library.<br>4. Only without external load balancer.<br>5. From worker nodes.<br>6. Only if SSL is not terminated at external load balancer.<br>7. Usually Ingress backed by infrastructure load balancer and/or nodeport.</td>
<td colspan="5">Notes:<br><br>1. Nodes running standalone server or Rancher HA deployment.<br>2. Only for hosted clusters.<br>3. Required to fetch Rancher chart library.<br>4. Only without external load balancer.<br>5. From worker nodes.<br>6. For direct access to the Kubernetes API without Rancher.<br>7. Usually Ingress backed by infrastructure load balancer and/or nodeport.</td>
</tr>
</tbody>
</table>
</table>
layouts/shortcodes/requirements_rollback.html
-15
View File
@@ -1,15 +0,0 @@
<div>
<ul>
<li>
<p>Rancher Kubernetes Engine v0.1.7 or later</p>
<p>The commands for taking <code>etcd</code> snapshots are only available in RKE v0.1.7 and later.</p>
</li>
<li>
<p><code>rancher-cluster.yml</code></p>
<p>You'll need the RKE config file that you used for Rancher install, <code>rancher-cluster.yml</code>. You created this file during your initial install. Place this file in same directory as the RKE binary.</p>
</li>
<li>
You must restore each of your <code>etcd</code> nodes to the <i>same</i> snapshot. Copy the snapshot you're using from one of your nodes to the others before running the <code>etcd snapshot-restore</code> command.
</li>
</ul>
</div>
static/img/rancher/port-communications.svg
+1 -1
View File
File diff suppressed because one or more lines are too long
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 21 KiB

After

Width:  |  Height:  |  Size: 21 KiB