mirror of
https://github.com/rancher/rancher-docs.git
synced 2026-05-29 16:15:30 +00:00
Catherine Luse
GitHub
@@ -0,0 +1,14 @@
|
||||
---
|
||||
name: Request a K3S change
|
||||
about: I'd like to request a change to the K3s documentation.
|
||||
title: "[K3s] "
|
||||
labels: K3s
|
||||
assignees: ''
|
||||
|
||||
---
|
||||
|
||||
**Request Summary:**
|
||||
<!-- Briefly describe what you would like to change in K3s documentation below this line. This can include any type of change such as deleting something from a section, a whole page, or adding a new section or page, or any updates/additions. -->
|
||||
|
||||
**Details:**
|
||||
<!-- If you have specific content you want changed, please indicate this below this line. -->
|
||||
@@ -0,0 +1,14 @@
|
||||
---
|
||||
name: Request a Rancher 2 change
|
||||
about: I'd like to request a change to the Rancher 2.x documentation.
|
||||
title: "[Rancher2] "
|
||||
labels: Rancher2
|
||||
assignees: ''
|
||||
|
||||
---
|
||||
|
||||
**Request Summary:**
|
||||
<!-- Briefly describe what you would like to change in Rancher 2.x documentation below this line. This can include any type of change such as deleting something from a section, a whole page, or adding a new section or page, or any updates/additions. -->
|
||||
|
||||
**Details:**
|
||||
<!-- If you have specific content you want changed, please indicate this below this line. -->
|
||||
@@ -0,0 +1,15 @@
|
||||
---
|
||||
name: Request something else
|
||||
about: I have a bug to report, or I'd like to request a change in documentation other
|
||||
than Rancher2 and K3s.
|
||||
title: ''
|
||||
labels: ''
|
||||
assignees: ''
|
||||
|
||||
---
|
||||
|
||||
**Summary:**
|
||||
<!-- Briefly describe what you would like to change in the documentation below this line. Or if you have a bug to report, please explain below this line. If a hyperlink is broken, please be specific and if you know where the link should instead route to, please indicate this. -->
|
||||
|
||||
**Details:**
|
||||
<!-- If you have any details to provide or need a specific change please indicate these details below this line. -->
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 55 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 179 KiB |
@@ -19,7 +19,7 @@ This section contains advanced information describing the different ways you can
|
||||
- [Additional preparation for Alpine Linux setup](#additional-preparation-for-alpine-linux-setup)
|
||||
- [Running K3d (K3s in Docker) and docker-compose](#running-k3d-k3s-in-docker-and-docker-compose)
|
||||
- [Enabling legacy iptables on Raspbian Buster](#enabling-legacy-iptables-on-raspbian-buster)
|
||||
- [Experimental SELinux Support](#experimental-selinux-support)
|
||||
- [SELinux Support](#selinux-support)
|
||||
|
||||
# Certificate Rotation
|
||||
|
||||
@@ -304,15 +304,20 @@ sudo update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy
|
||||
sudo reboot
|
||||
```
|
||||
|
||||
# Experimental SELinux Support
|
||||
# SELinux Support
|
||||
|
||||
As of release v1.17.4+k3s1, experimental support for SELinux has been added to K3s's embedded containerd. If you are installing K3s on a system where SELinux is enabled by default (such as CentOS), you must ensure the proper SELinux policies have been installed.
|
||||
_Supported as of v1.19.4+k3s1. Experimental as of v1.17.4+k3s1._
|
||||
|
||||
If you are installing K3s on a system where SELinux is enabled by default (such as CentOS), you must ensure the proper SELinux policies have been installed.
|
||||
|
||||
### Automatic Installation
|
||||
|
||||
_Available as of v1.19.3+k3s2_
|
||||
|
||||
The [install script]({{<baseurl>}}/k3s/latest/en/installation/install-options/#installation-script-options) will automatically install the SELinux RPM from the Rancher RPM repository if on a compatible system if not performing an air-gapped install. Automatic installation can be skipped by setting `INSTALL_K3S_SKIP_SELINUX_RPM=true`.
|
||||
|
||||
### Manual Installation
|
||||
|
||||
{{% tabs %}}
|
||||
{{% tab "automatic installation" %}}
|
||||
As of release v1.19.3+k3s2, the [install script]({{<baseurl>}}/k3s/latest/en/installation/install-options/#installation-script-options) will automatically install the SELinux RPM from the Rancher RPM repository if on a compatible system if not performing an air-gapped install. Automatic installation can be skipped by setting `INSTALL_K3S_SKIP_SELINUX_RPM=true`.
|
||||
{{% /tab %}}
|
||||
{{% tab "manual installation" %}}
|
||||
The necessary policies can be installed with the following commands:
|
||||
```
|
||||
yum install -y container-selinux selinux-policy-base
|
||||
@@ -321,17 +326,14 @@ yum install -y https://rpm.rancher.io/k3s/latest/common/centos/7/noarch/k3s-seli
|
||||
|
||||
To force the install script to log a warning rather than fail, you can set the following environment variable: `INSTALL_K3S_SELINUX_WARN=true`.
|
||||
|
||||
{{% /tab %}}
|
||||
{{% /tabs %}}
|
||||
### Enabling and Disabling SELinux Enforcement
|
||||
|
||||
The way that SELinux enforcement is enabled or disabled depends on the K3s version. Prior to v1.19.x, SELinux enablement for the builtin containerd was automatic but could be disabled by passing `--disable-selinux`. With v1.19.x and beyond, enabling SELinux must be affirmatively configured via the `--selinux` flag or config file entry. Servers and agents that specify both the `--selinux` and (deprecated) `--disable-selinux` flags will fail to start.
|
||||
|
||||
Using a custom `--data-dir` under SELinux is not supported. To customize it, you would most likely need to write your own custom policy. For guidance, you could refer to the [containers/container-selinux](https://github.com/containers/container-selinux) repository, which contains the SELinux policy files for Container Runtimes, and the [rancher/k3s-selinux](https://github.com/rancher/k3s-selinux) repository, which contains the SELinux policy for K3s .
|
||||
The way that SELinux enforcement is enabled or disabled depends on the K3s version.
|
||||
|
||||
{{% tabs %}}
|
||||
{{% tab "K3s v1.19.1+k3s1" %}}
|
||||
|
||||
To leverage experimental SELinux, specify the `--selinux` flag when starting K3s servers and agents.
|
||||
To leverage SELinux, specify the `--selinux` flag when starting K3s servers and agents.
|
||||
|
||||
This option can also be specified in the K3s [configuration file:]({{<baseurl>}}/k3s/latest/en/installation/install-options/#configuration-file)
|
||||
|
||||
@@ -341,12 +343,16 @@ selinux: true
|
||||
|
||||
The `--disable-selinux` option should not be used. It is deprecated and will be either ignored or will be unrecognized, resulting in an error, in future minor releases.
|
||||
|
||||
Using a custom `--data-dir` under SELinux is not supported. To customize it, you would most likely need to write your own custom policy. For guidance, you could refer to the [containers/container-selinux](https://github.com/containers/container-selinux) repository, which contains the SELinux policy files for Container Runtimes, and the [rancher/k3s-selinux](https://github.com/rancher/k3s-selinux) repository, which contains the SELinux policy for K3s .
|
||||
|
||||
{{%/tab%}}
|
||||
{{% tab "K3s prior to v1.19.1+k3s1" %}}
|
||||
|
||||
You can turn off SELinux enforcement in the embedded containerd by launching K3s with the `--disable-selinux` flag.
|
||||
SELinux is automatically enabled for the built-in containerd.
|
||||
|
||||
To turn off SELinux enforcement in the embedded containerd, launch K3s with the `--disable-selinux` flag.
|
||||
|
||||
Using a custom `--data-dir` under SELinux is not supported. To customize it, you would most likely need to write your own custom policy. For guidance, you could refer to the [containers/container-selinux](https://github.com/containers/container-selinux) repository, which contains the SELinux policy files for Container Runtimes, and the [rancher/k3s-selinux](https://github.com/rancher/k3s-selinux) repository, which contains the SELinux policy for K3s .
|
||||
|
||||
{{%/tab%}}
|
||||
{{% /tabs %}}
|
||||
|
||||
Note that support for SELinux in containerd is still under development. Progress can be tracked in [this pull request](https://github.com/containerd/cri/pull/1246).
|
||||
|
||||
@@ -60,6 +60,7 @@ spec:
|
||||
| spec.helmVersion | v3 | Helm version to use (`v2` or `v3`) | |
|
||||
| spec.bootstrap | False | Set to True if this chart is needed to bootstrap the cluster (Cloud Controller Manager, etc) | |
|
||||
| spec.set | | Override simple default Chart values. These take precedence over options set via valuesContent. | `--set` / `--set-string` |
|
||||
| spec.jobImage | | Specify the image to use when installing the helm chart. E.g. rancher/klipper-helm:v0.3.0 . | |
|
||||
| spec.valuesContent | | Override complex default Chart values via YAML file content | `--values` |
|
||||
| spec.chartContent | | Base64-encoded chart archive .tgz - overrides spec.chart | CHART |
|
||||
|
||||
|
||||
@@ -33,7 +33,7 @@ sudo mkdir -p /var/lib/rancher/k3s/agent/images/
|
||||
sudo cp ./k3s-airgap-images-$ARCH.tar /var/lib/rancher/k3s/agent/images/
|
||||
```
|
||||
|
||||
Place the k3s binary at /usr/local/bin/k3s and ensure it is executable.
|
||||
Place the k3s binary at `/usr/local/bin/k3s` and ensure it is executable.
|
||||
|
||||
Follow the steps in the next section to install K3s.
|
||||
|
||||
|
||||
@@ -79,6 +79,8 @@ For details on configuring the K3s agent, refer to the [agent configuration refe
|
||||
|
||||
### Configuration File
|
||||
|
||||
_Available as of v1.19.1+k3s1_
|
||||
|
||||
In addition to configuring K3s with environment variables and CLI arguments, K3s can also use a config file.
|
||||
|
||||
By default, values present in a YAML file located at `/etc/rancher/k3s/config.yaml` will be used on install.
|
||||
|
||||
@@ -17,17 +17,14 @@ If all your nodes have the same hostname, use the `--with-node-id` option to app
|
||||
|
||||
## Operating Systems
|
||||
|
||||
K3s should run on just about any flavor of Linux.
|
||||
K3s is expected to work on most modern Linux systems.
|
||||
|
||||
K3s is officially supported and tested on the following operating systems and their subsequent non-major releases:
|
||||
Some OSs have specific requirements:
|
||||
|
||||
* Ubuntu 16.04 (amd64)
|
||||
* Ubuntu 18.04 (amd64)
|
||||
* Raspbian Buster*
|
||||
- If you are using **Raspbian Buster**, follow [these steps]({{<baseurl>}}/k3s/latest/en/advanced/#enabling-legacy-iptables-on-raspbian-buster) to switch to legacy iptables.
|
||||
- If you are using **Alpine Linux**, follow [these steps]({{<baseurl>}}/k3s/latest/en/advanced/#additional-preparation-for-alpine-linux-setup) for additional setup.
|
||||
|
||||
\* If you are using **Raspbian Buster**, follow [these steps]({{<baseurl>}}/k3s/latest/en/advanced/#enabling-legacy-iptables-on-raspbian-buster) to switch to legacy iptables.
|
||||
|
||||
If you are using **Alpine Linux**, follow [these steps]({{<baseurl>}}/k3s/latest/en/advanced/#additional-preparation-for-alpine-linux-setup) for additional setup.
|
||||
For more information on which OSs were tested with Rancher managed K3s clusters, refer to the [Rancher support and maintenance terms.](https://rancher.com/support-maintenance-terms/)
|
||||
|
||||
## Hardware
|
||||
|
||||
@@ -36,6 +33,8 @@ Hardware requirements scale based on the size of your deployments. Minimum recom
|
||||
* RAM: 512MB Minimum (we recommend at least 1GB)
|
||||
* CPU: 1 Minimum
|
||||
|
||||
[This section](./resource-profiling) captures the results of tests to determine minimum resource requirements for the K3s agent, the K3s server with a workload, and the K3s server with one agent. It also contains analysis about what has the biggest impact on K3s server and agent utilization, and how the cluster datastore can be protected from interference from agents and workloads.
|
||||
|
||||
#### Disks
|
||||
|
||||
K3s performance depends on the performance of the database. To ensure optimal speed, we recommend using an SSD when possible. Disk performance will vary on ARM devices utilizing an SD card or eMMC.
|
||||
|
||||
+139
@@ -0,0 +1,139 @@
|
||||
---
|
||||
title: K3s Resource Profiling
|
||||
shortTitle: Resource Profiling
|
||||
weight: 1
|
||||
---
|
||||
|
||||
This section captures the results of tests to determine minimum resource requirements for K3s.
|
||||
|
||||
The results are summarized as follows:
|
||||
|
||||
| Components | Processor | Min CPU | Min RAM with Kine/SQLite | Min RAM with Embedded etcd |
|
||||
|------------|-----|----------|-------------------------|---------------------------|
|
||||
| K3s server with a workload | Intel(R) Xeon(R) Platinum 8124M CPU, 3.00 GHz | 10% of a core | 768 M | 896 M |
|
||||
| K3s cluster with a single agent | Intel(R) Xeon(R) Platinum 8124M CPU, 3.00 GHz | 10% of a core | 512 M | 768 M |
|
||||
| K3s agent | Intel(R) Xeon(R) Platinum 8124M CPU, 3.00 GHz | 5% of a core | 256 M | 256 M |
|
||||
| K3s server with a workload | Pi4B BCM2711, 1.50 GHz | 20% of a core | 768 M | 896 M |
|
||||
| K3s cluster with a single agent | Pi4B BCM2711, 1.50 GHz | 20% of a core | 512 M | 768 M |
|
||||
| K3s agent | Pi4B BCM2711, 1.50 GHz | 10% of a core | 256 M | 256 M |
|
||||
|
||||
- [Scope of Resource Testing](#scope-of-resource-testing)
|
||||
- [Components Included for Baseline Measurements](#components-included-for-baseline-measurements)
|
||||
- [Methodology](#methodology)
|
||||
- [Environment](#environment)
|
||||
- [Baseline Resource Requirements](#baseline-resource-requirements)
|
||||
- [K3s Server with a Workload](#k3s-server-with-a-workload)
|
||||
- [K3s Cluster with a Single Agent](#k3s-cluster-with-a-single-agent)
|
||||
- [K3s Agent](#k3s-agent)
|
||||
- [Analysis](#analysis)
|
||||
- [Primary Resource Utilization Drivers](#primary-resource-utilization-drivers)
|
||||
- [Preventing Agents and Workloads from Interfering with the Cluster Datastore](#preventing-agents-and-workloads-from-interfering-with-the-cluster-datastore)
|
||||
|
||||
# Scope of Resource Testing
|
||||
|
||||
The resource tests were intended to address the following problem statements:
|
||||
|
||||
- On a single-node cluster, determine the legitimate minimum amount of CPU, memory, and IOPs that should be set aside to run the entire K3s stack server stack, assuming that a real workload will be deployed on the cluster.
|
||||
- On an agent (worker) node, determine the legitimate minimum amount of CPU, memory, and IOPs that should be set aside for the Kubernetes and K3s control plane components (the kubelet and k3s agent).
|
||||
|
||||
# Components Included for Baseline Measurements
|
||||
|
||||
The tested components are:
|
||||
|
||||
* K3s 1.19.2 with all packaged components enabled
|
||||
* Prometheus + Grafana monitoring stack
|
||||
* Kubernetes Example PHP Guestbook app
|
||||
|
||||
These are baseline figures for a stable system using only K3s packaged components (Traefik Ingress, Klipper lb, local-path storage) running a standard monitoring stack (Prometheus and Grafana) and the Guestbook example app.
|
||||
|
||||
Resource figures including IOPS are for the Kubernetes datastore and control plane only, and do not include overhead for system-level management agents or logging, container image management, or any workload-specific requirements.
|
||||
|
||||
# Methodology
|
||||
|
||||
A standalone instance of Prometheus v2.21.0 was used to collect host CPU, memory, and disk IO statistics using `prometheus-node-exporter` installed via apt.
|
||||
|
||||
`systemd-cgtop` was used to spot-check systemd cgroup-level CPU and memory utilization. `system.slice/k3s.service` tracks resource utilization for both K3s and containerd, while individual pods are under the `kubepods` hierarchy.
|
||||
|
||||
Additional detailed K3s memory utilization data was collected from the `process_resident_memory_bytes` and `go_memstats_alloc_bytes` metrics using the kubelet exporter integrated into the server and agent processes.
|
||||
|
||||
Utilization figures were based on 95th percentile readings from steady state operation on nodes running the described workloads.
|
||||
|
||||
# Environment
|
||||
|
||||
OS: Ubuntu 20.04 x86_64, aarch64
|
||||
|
||||
Hardware:
|
||||
|
||||
- AWS c5d.xlarge - 4 core, 8 GB RAM, NVME SSD
|
||||
- Raspberry Pi 4 Model B - 4 core, 8 GB RAM, Class 10 SDHC
|
||||
|
||||
# Baseline Resource Requirements
|
||||
|
||||
This section captures the results of tests to determine minimum resource requirements for the K3s agent, the K3s server with a workload, and the K3s server with one agent.
|
||||
|
||||
### K3s Server with a Workload
|
||||
|
||||
These are the requirements for a single-node cluster in which the K3s server shares resources with a workload.
|
||||
|
||||
The CPU requirements are:
|
||||
|
||||
| Resource Requirement | Tested Processor |
|
||||
|-----------|-----------------|
|
||||
| 10% of a core | Intel(R) Xeon(R) Platinum 8124M CPU, 3.00 GHz |
|
||||
| 20% of a core | Low-power processor such as Pi4B BCM2711, 1.50 GHz |
|
||||
|
||||
The IOPS and memory requirements are:
|
||||
|
||||
| Tested Datastore | IOPS | KiB/sec | Latency | RAM |
|
||||
|-----------|------|---------|---------|--------|
|
||||
| Kine/SQLite | 10 | 500 | < 10 ms | 768 M |
|
||||
| Embedded etcd | 50 | 250 | < 5 ms | 896 M |
|
||||
|
||||
### K3s Cluster with a Single Agent
|
||||
|
||||
These are the baseline requirements for a K3s cluster with a K3s server node and a K3s agent, but no workload.
|
||||
|
||||
The CPU requirements are:
|
||||
|
||||
| Resource Requirement | Tested Processor |
|
||||
|-----------|-----------------|
|
||||
| 10% of a core | Intel(R) Xeon(R) Platinum 8124M CPU, 3.00 GHz |
|
||||
| 20% of a core | Pi4B BCM2711, 1.50 GHz |
|
||||
|
||||
The IOPS and memory requirements are:
|
||||
|
||||
| Datastore | IOPS | KiB/sec | Latency | RAM |
|
||||
|-----------|------|---------|---------|--------|
|
||||
| Kine/SQLite | 10 | 500 | < 10 ms | 512 M |
|
||||
| Embedded etcd | 50 | 250 | < 5 ms | 768 M |
|
||||
|
||||
### K3s Agent
|
||||
|
||||
The CPU requirements are:
|
||||
|
||||
Resource Requirement | Tested Processor |
|
||||
|-----------|-----------------|
|
||||
| 5% of a core | Intel(R) Xeon(R) Platinum 8124M CPU, 3.00 GHz |
|
||||
| 10% of a core | Pi4B BCM2711, 1.50 GHz |
|
||||
|
||||
256 M of RAM is required.
|
||||
|
||||
# Analysis
|
||||
|
||||
This section captures what has the biggest impact on K3s server and agent utilization, and how the cluster datastore can be protected from interference from agents and workloads.
|
||||
|
||||
### Primary Resource Utilization Drivers
|
||||
|
||||
K3s server utilization figures are primarily driven by support of the Kubernetes datastore (kine or etcd), API Server, Controller-Manager, and Scheduler control loops, as well as any management tasks necessary to effect changes to the state of the system. Operations that place additional load on the Kubernetes control plane, such as creating/modifying/deleting resources, will cause temporary spikes in utilization. Using operators or apps that make extensive use of the Kubernetes datastore (such as Rancher or other Operator-type applications) will increase the server's resource requirements. Scaling up the cluster by adding additional nodes or creating many cluster resources will increase the server's resource requirements.
|
||||
|
||||
K3s agent utilization figures are primarily driven by support of container lifecycle management control loops. Operations that involve managing images, provisioning storage, or creating/destroying containers will cause temporary spikes in utilization. Image pulls in particular are typically highly CPU and IO bound, as they involve decompressing image content to disk. If possible, workload storage (pod ephemeral storage and volumes) should be isolated from the agent components (/var/lib/rancher/k3s/agent) to ensure that there are no resource conflicts.
|
||||
|
||||
### Preventing Agents and Workloads from Interfering with the Cluster Datastore
|
||||
|
||||
When running in an environment where the server is also hosting workload pods, care should be taken to ensure that agent and workload IOPS do not interfere with the datastore.
|
||||
|
||||
This can be best accomplished by placing the server components (/var/lib/rancher/k3s/server) on a different storage medium than the agent components (/var/lib/rancher/k3s/agent), which include the containerd image store.
|
||||
|
||||
Workload storage (pod ephemeral storage and volumes) should also be isolated from the datastore.
|
||||
|
||||
Failure to meet datastore throughput and latency requirements may result in delayed response from the control plane and/or failure of the control plane to maintain system state.
|
||||
@@ -40,7 +40,7 @@ Apply the Canal YAML.
|
||||
Ensure the settings were applied by running the following command on the host:
|
||||
|
||||
```
|
||||
cat /etc/cni/net.d/10-calico.conflist
|
||||
cat /etc/cni/net.d/10-canal.conflist
|
||||
```
|
||||
|
||||
You should see that IP forwarding is set to true.
|
||||
@@ -61,7 +61,7 @@ Apply the Calico YAML.
|
||||
Ensure the settings were applied by running the following command on the host:
|
||||
|
||||
```
|
||||
cat /etc/cni/net.d/10-canal.conflist
|
||||
cat /etc/cni/net.d/10-calico.conflist
|
||||
```
|
||||
|
||||
You should see that IP forwarding is set to true.
|
||||
|
||||
@@ -89,12 +89,6 @@ kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/master/depl
|
||||
|
||||
Longhorn will be installed in the namespace `longhorn-system`.
|
||||
|
||||
Before we create a PVC, we will create a storage class for Longhorn with this yaml:
|
||||
|
||||
```
|
||||
kubectl create -f https://raw.githubusercontent.com/longhorn/longhorn/master/examples/storageclass.yaml
|
||||
```
|
||||
|
||||
Apply the yaml to create the PVC and pod:
|
||||
|
||||
```
|
||||
|
||||
@@ -17,7 +17,7 @@ If your organization uses Keycloak Identity Provider (IdP) for user authenticati
|
||||
`Sign Documents` | `ON` <sup>1</sup>
|
||||
`Sign Assertions` | `ON` <sup>1</sup>
|
||||
All other `ON/OFF` Settings | `OFF`
|
||||
`Client ID` | `https://yourRancherHostURL/v1-saml/keycloak/saml/metadata`<sup>2</sup>
|
||||
`Client ID` | Either `https://yourRancherHostURL/v1-saml/keycloak/saml/metadata` or the value configured in the `Entry ID Field` of the Rancher Keycloak configuration<sup>2</sup>
|
||||
`Client Name` | <CLIENT_NAME> (e.g. `rancher`)
|
||||
`Client Protocol` | `SAML`
|
||||
`Valid Redirect URI` | `https://yourRancherHostURL/v1-saml/keycloak/saml/acs`
|
||||
@@ -65,15 +65,16 @@ If your organization uses Keycloak Identity Provider (IdP) for user authenticati
|
||||
1. Complete the **Configure Keycloak Account** form.
|
||||
|
||||
|
||||
| Field | Description |
|
||||
| ------------------------- | -------------------------------------------------------------------------------------- |
|
||||
| Display Name Field | The attribute that contains the display name of users. <br/><br/>Example: `givenName` |
|
||||
| User Name Field | The attribute that contains the user name/given name. <br/><br/>Example: `email` |
|
||||
| UID Field | An attribute that is unique to every user. <br/><br/>Example: `email` |
|
||||
| Groups Field | Make entries for managing group memberships. <br/><br/>Example: `member` |
|
||||
| Rancher API Host | The URL for your Rancher Server. |
|
||||
| Private Key / Certificate | A key/certificate pair to create a secure shell between Rancher and your IdP. |
|
||||
| IDP-metadata | The `metadata.xml` file that you exported from your IdP server. |
|
||||
| Field | Description |
|
||||
| ------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| Display Name Field | The attribute that contains the display name of users. <br/><br/>Example: `givenName` |
|
||||
| User Name Field | The attribute that contains the user name/given name. <br/><br/>Example: `email` |
|
||||
| UID Field | An attribute that is unique to every user. <br/><br/>Example: `email` |
|
||||
| Groups Field | Make entries for managing group memberships. <br/><br/>Example: `member` |
|
||||
| Entity ID Field | The ID that needs to be configured as a client ID in the Keycloak client. <br/><br/>Default: `https://yourRancherHostURL/v1-saml/keycloak/saml/metadata` |
|
||||
| Rancher API Host | The URL for your Rancher Server. |
|
||||
| Private Key / Certificate | A key/certificate pair to create a secure shell between Rancher and your IdP. |
|
||||
| IDP-metadata | The `metadata.xml` file that you exported from your IdP server. |
|
||||
|
||||
>**Tip:** You can generate a key/certificate pair using an openssl command. For example:
|
||||
>
|
||||
|
||||
+2
-2
@@ -338,8 +338,8 @@ Documented here is a minimum set of permissions necessary to use all functionali
|
||||
|
||||
Resource | Description
|
||||
---------|------------
|
||||
Service Role | The service role provides Kubernetes the permissions it requires to manage resources on your behalf. Rancher can create the service role with the following [Service Role Permissions](http://localhost:9001/rancher/v2.x/en/cluster-provisioning/hosted-kubernetes-clusters/eks/#service-role-permissions).
|
||||
VPC | Provides isolated network resouces utilised by EKS and worker nodes. Rancher can create the VPC resouces with the follwoing [VPC Permissions](http://localhost:9001/rancher/v2.x/en/cluster-provisioning/hosted-kubernetes-clusters/eks/#vpc-permissions).
|
||||
Service Role | The service role provides Kubernetes the permissions it requires to manage resources on your behalf. Rancher can create the service role with the following [Service Role Permissions]({{<baseurl>}}/rancher/v2.x/en/cluster-provisioning/hosted-kubernetes-clusters/eks/#service-role-permissions).
|
||||
VPC | Provides isolated network resources utilised by EKS and worker nodes. Rancher can create the VPC resources with the following [VPC Permissions]({{<baseurl>}}/rancher/v2.x/en/cluster-provisioning/hosted-kubernetes-clusters/eks/#vpc-permissions).
|
||||
|
||||
|
||||
Resource targeting uses `*` as the ARN of many of the resources created cannot be known prior to creating the EKS cluster in Rancher.
|
||||
|
||||
@@ -68,6 +68,9 @@ For **Host Gateway (L2bridge)** networking, it's best to use the same Layer 2 ne
|
||||
|
||||
For **VXLAN (Overlay)** networking, the [KB4489899](https://support.microsoft.com/en-us/help/4489899) hotfix must be installed. Most cloud-hosted VMs already have this hotfix.
|
||||
|
||||
If you are configuring DHCP options sets for an AWS virtual private cloud, note that in the `domain-name` option field, only one domain name can be specified. According to the DHCP options [documentation:](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_DHCP_Options.html)
|
||||
|
||||
> Some Linux operating systems accept multiple domain names separated by spaces. However, other Linux operating systems and Windows treat the value as a single domain, which results in unexpected behavior. If your DHCP options set is associated with a VPC that has instances with multiple operating systems, specify only one domain name.
|
||||
### Architecture Requirements
|
||||
|
||||
The Kubernetes cluster management nodes (`etcd` and `controlplane`) must be run on Linux nodes.
|
||||
|
||||
@@ -20,7 +20,9 @@ The charts page contains all Rancher, Partner, and Custom Charts.
|
||||
* Partner charts reside under the Partners label
|
||||
* Custom charts will show up under the name of the repository
|
||||
|
||||
All three types are deployed and managed in the same way.
|
||||
All three types are deployed and managed in the same way.
|
||||
|
||||
> Apps managed by the Cluster Manager should continue to be managed only by the Cluster Manager, and apps managed with the Cluster Explorer must be managed only by the Cluster Explorer.
|
||||
|
||||
### Repositories
|
||||
|
||||
@@ -29,17 +31,17 @@ From the left sidebar select _"Repositories"_.
|
||||
These items represent helm repositories, and can be either traditional helm endpoints which have an index.yaml, or git repositories which will be cloned and can point to a specific branch. In order to use custom charts, simply add your repository here and they will become available in the Charts tab under the name of the repository.
|
||||
|
||||
|
||||
### Helm compatilbitiy
|
||||
### Helm Compatibility
|
||||
|
||||
The Cluster Explorer only supports Helm 3 compatible charts.
|
||||
|
||||
|
||||
### Deployment and Upgrades
|
||||
|
||||
From the _"Charts"_ tab select a Chart to install. Rancher and Partner charts may have extra configurations available through custom pages or questions.yaml files, but all chart installations can modify the values.yaml and other basic settings. Once you click install, a helm operation job is deployed, and the console for the job is displayed.
|
||||
From the _"Charts"_ tab select a Chart to install. Rancher and Partner charts may have extra configurations available through custom pages or questions.yaml files, but all chart installations can modify the values.yaml and other basic settings. Once you click install, a Helm operation job is deployed, and the console for the job is displayed.
|
||||
|
||||
To view all recent changes, go to the _"Recent Operations"_ tab. From there you can view the call that was made, conditions, events, and logs.
|
||||
|
||||
After installing a chart, you can find it in the _"Installed Apps"_ tab. In this section you can upgrade or delete the installation, and see further details. When choosing to upgrade, the form and values presented will be the same as installation.
|
||||
After installing a chart, you can find it in the _"Installed Apps"_ tab. In this section you can upgrade or delete the installation, and see further details. When choosing to upgrade, the form and values presented will be the same as installation.
|
||||
|
||||
Most Rancher tools have additional pages located in the toolbar below the _"Apps & Marketplace"_ section to help manage and use the features. These pages include links to dashboards, forms to easily add Custom Resources, and additional information.
|
||||
|
||||
@@ -109,7 +109,7 @@ These instructions are adapted from the [official cert-manager documentation](ht
|
||||
|
||||
```
|
||||
# Install the CustomResourceDefinition resources separately
|
||||
kubectl apply --validate=false -f https://github.com/jetstack/cert-manager/releases/download/v0.15.0/cert-manager.crds.yaml
|
||||
kubectl apply --validate=false -f https://github.com/jetstack/cert-manager/releases/download/v1.0.4/cert-manager.crds.yaml
|
||||
|
||||
# **Important:**
|
||||
# If you are running Kubernetes v1.15 or below, you
|
||||
@@ -134,7 +134,7 @@ helm repo update
|
||||
helm install \
|
||||
cert-manager jetstack/cert-manager \
|
||||
--namespace cert-manager \
|
||||
--version v0.15.0
|
||||
--version v1.0.4
|
||||
```
|
||||
|
||||
Once you’ve installed cert-manager, you can verify it is deployed correctly by checking the cert-manager namespace for running pods:
|
||||
|
||||
+4
-3
@@ -85,12 +85,12 @@ By default, Rancher generates a CA and uses cert-manager to issue the certificat
|
||||
1. Fetch the latest cert-manager chart available from the [Helm chart repository](https://hub.helm.sh/charts/jetstack/cert-manager).
|
||||
|
||||
```plain
|
||||
helm fetch jetstack/cert-manager --version v0.12.0
|
||||
helm fetch jetstack/cert-manager --version v1.0.4
|
||||
```
|
||||
|
||||
1. Render the cert manager template with the options you would like to use to install the chart. Remember to set the `image.repository` option to pull the image from your private registry. This will create a `cert-manager` directory with the Kubernetes manifest files.
|
||||
```plain
|
||||
helm template cert-manager ./cert-manager-v0.12.0.tgz --output-dir . \
|
||||
helm template cert-manager ./cert-manager-v1.0.4.tgz --output-dir . \
|
||||
--namespace cert-manager \
|
||||
--set image.repository=<REGISTRY.YOURDOMAIN.COM:PORT>/quay.io/jetstack/cert-manager-controller \
|
||||
--set webhook.image.repository=<REGISTRY.YOURDOMAIN.COM:PORT>/quay.io/jetstack/cert-manager-webhook \
|
||||
@@ -99,8 +99,9 @@ By default, Rancher generates a CA and uses cert-manager to issue the certificat
|
||||
|
||||
1. Download the required CRD file for cert-manager
|
||||
```plain
|
||||
curl -L -o cert-manager/cert-manager-crd.yaml https://raw.githubusercontent.com/jetstack/cert-manager/release-0.12/deploy/manifests/00-crds.yaml
|
||||
curl -L -o cert-manager/cert-manager-crd.yaml https://github.com/jetstack/cert-manager/releases/download/v1.0.4/cert-manager.crds.yaml
|
||||
```
|
||||
|
||||
1. Render the Rancher template, declaring your chosen options. Use the reference table below to replace each placeholder. Rancher needs to be configured to use the private registry in order to provision any Rancher launched Kubernetes clusters or Rancher tools.
|
||||
|
||||
|
||||
|
||||
+1
-1
@@ -63,7 +63,7 @@ In a Kubernetes Install, if you elect to use the Rancher default self-signed TLS
|
||||
```plain
|
||||
helm repo add jetstack https://charts.jetstack.io
|
||||
helm repo update
|
||||
helm fetch jetstack/cert-manager --version v0.12.0
|
||||
helm fetch jetstack/cert-manager --version v1.0.4
|
||||
helm template ./cert-manager-<version>.tgz | grep -oP '(?<=image: ").*(?=")' >> ./rancher-images.txt
|
||||
```
|
||||
|
||||
|
||||
+1
-1
@@ -34,7 +34,7 @@ helm upgrade --install cert-manager jetstack/cert-manager \
|
||||
--namespace cert-manager --version v0.15.2 \
|
||||
--set http_proxy=http://${proxy_host} \
|
||||
--set https_proxy=http://${proxy_host} \
|
||||
--set no_proxy=127.0.0.0/8\\,10.0.0.0/8\\,172.16.0.0/12\\,192.168.0.0/16
|
||||
--set no_proxy=127.0.0.0/8\\,10.0.0.0/8\\,172.16.0.0/12\\,192.168.0.0/16\\,.svc\\,.cluster.local
|
||||
```
|
||||
|
||||
Now you should wait until cert-manager is finished starting up:
|
||||
|
||||
+1
-1
@@ -42,7 +42,7 @@ sudo usermod -aG docker YOUR_USERNAME
|
||||
And configure the Docker daemon to use the proxy to pull images:
|
||||
|
||||
```
|
||||
sudo mkdir -p mkdir /etc/systemd/system/docker.service.d
|
||||
sudo mkdir -p /etc/systemd/system/docker.service.d
|
||||
cat <<'EOF' | sudo tee /etc/systemd/system/docker.service.d/http-proxy.conf > /dev/null
|
||||
[Service]
|
||||
Environment="HTTP_PROXY=http://${proxy_host}"
|
||||
|
||||
+13
-1
@@ -83,7 +83,19 @@ If you are visiting this page to complete an air gap installation, you must prep
|
||||
|
||||
### Persistent Data
|
||||
|
||||
{{< persistentdata >}}
|
||||
Rancher uses etcd as a datastore. When Rancher is installed with Docker, the embedded etcd is being used. The persistent data is at the following path in the container: `/var/lib/rancher`.
|
||||
|
||||
You can bind mount a host volume to this location to preserve data on the host it is running on:
|
||||
|
||||
```
|
||||
docker run -d --restart=unless-stopped \
|
||||
-p 80:80 -p 443:443 \
|
||||
-v /opt/rancher:/var/lib/rancher \
|
||||
--privileged \
|
||||
rancher/rancher:latest
|
||||
```
|
||||
|
||||
As of Rancher v2.5, privileged access is [required.](../#privileged-access-for-rancher-v2-5)
|
||||
|
||||
### Running `rancher/rancher` and `rancher/rancher-agent` on the Same Node
|
||||
|
||||
|
||||
+3
-1
@@ -26,6 +26,8 @@ Passing environment variables to the Rancher container can be done using `-e KEY
|
||||
- `127.0.0.1`
|
||||
- `0.0.0.0`
|
||||
- `10.0.0.0/8`
|
||||
- `.svc`
|
||||
- `.cluster.local`
|
||||
|
||||
The example below is based on a proxy server accessible at `http://192.168.0.1:3128`, and excluding usage the proxy when accessing network range `192.168.10.0/24` and every hostname under the domain `example.com`.
|
||||
|
||||
@@ -34,7 +36,7 @@ docker run -d --restart=unless-stopped \
|
||||
-p 80:80 -p 443:443 \
|
||||
-e HTTP_PROXY="http://192.168.10.1:3128" \
|
||||
-e HTTPS_PROXY="http://192.168.10.1:3128" \
|
||||
-e NO_PROXY="localhost,127.0.0.1,0.0.0.0,10.0.0.0/8,192.168.10.0/24,example.com" \
|
||||
-e NO_PROXY="localhost,127.0.0.1,0.0.0.0,10.0.0.0/8,192.168.10.0/24,.svc,.cluster.local,example.com" \
|
||||
--privileged \
|
||||
rancher/rancher:latest
|
||||
```
|
||||
|
||||
+13
-1
@@ -197,7 +197,19 @@ If you are visiting this page to complete an [Air Gap Installation]({{<baseurl>}
|
||||
|
||||
### Persistent Data
|
||||
|
||||
{{< persistentdata >}}
|
||||
Rancher uses etcd as a datastore. When Rancher is installed with Docker, the embedded etcd is being used. The persistent data is at the following path in the container: `/var/lib/rancher`.
|
||||
|
||||
You can bind mount a host volume to this location to preserve data on the host it is running on:
|
||||
|
||||
```
|
||||
docker run -d --restart=unless-stopped \
|
||||
-p 80:80 -p 443:443 \
|
||||
-v /opt/rancher:/var/lib/rancher \
|
||||
--privileged \
|
||||
rancher/rancher:latest
|
||||
```
|
||||
|
||||
As of Rancher v2.5, privileged access is [required.](../#privileged-access-for-rancher-v2-5)
|
||||
|
||||
This layer 7 NGINX configuration is tested on NGINX version 1.13 (mainline) and 1.14 (stable).
|
||||
|
||||
|
||||
@@ -54,7 +54,7 @@ aliases:
|
||||
| `ingress.configurationSnippet` | "" | `string` - Add additional Nginx configuration. Can be used for proxy configuration. _Note: Available as of v2.0.15, v2.1.10 and v2.2.4_ |
|
||||
| `letsEncrypt.ingress.class` | "" | `string` - optional ingress class for the cert-manager acmesolver ingress that responds to the Let's Encrypt ACME challenges |
|
||||
| `proxy` | "" | `string` - HTTP[S] proxy server for Rancher |
|
||||
| `noProxy` | "127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16" | `string` - comma separated list of hostnames or ip address not to use the proxy |
|
||||
| `noProxy` | "127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,.svc,.cluster.local" | `string` - comma separated list of hostnames or ip address not to use the proxy |
|
||||
| `resources` | {} | `map` - rancher pod resource requests & limits |
|
||||
| `rancherImage` | "rancher/rancher" | `string` - rancher image source |
|
||||
| `rancherImageTag` | same as chart version | `string` - rancher/rancher image tag |
|
||||
@@ -134,11 +134,11 @@ Example on setting a static proxy header with `ingress.configurationSnippet`. Th
|
||||
|
||||
Rancher requires internet access for some functionality (helm charts). Use `proxy` to set your proxy server.
|
||||
|
||||
Add your IP exceptions to the `noProxy` list. Make sure you add the Service cluster IP range (default: 10.43.0.1/16) and any worker cluster `controlplane` nodes. Rancher supports CIDR notation ranges in this list.
|
||||
Add your IP exceptions to the `noProxy` list. Make sure you add the Pod cluster IP range (default: `10.42.0.0/16`), Service cluster IP range (default: `10.43.0.0/16`), the internal cluster domains (default: `.svc,.cluster.local`) and any worker cluster `controlplane` nodes. Rancher supports CIDR notation ranges in this list.
|
||||
|
||||
```plain
|
||||
--set proxy="http://<username>:<password>@<proxy_url>:<proxy_port>/"
|
||||
--set noProxy="127.0.0.0/8\,10.0.0.0/8\,172.16.0.0/12\,192.168.0.0/16"
|
||||
--set noProxy="127.0.0.0/8\,10.0.0.0/8\,172.16.0.0/12\,192.168.0.0/16,.svc,.cluster.local"
|
||||
```
|
||||
|
||||
### Additional Trusted CAs
|
||||
|
||||
+3
-3
@@ -53,16 +53,16 @@ In order to upgrade cert-manager, follow these instructions:
|
||||
helm uninstall cert-manager
|
||||
```
|
||||
|
||||
Delete the CustomResourceDefinition using the link to the version vX.Y you installed
|
||||
Delete the CustomResourceDefinition using the link to the version vX.Y.Z you installed
|
||||
|
||||
```plain
|
||||
kubectl delete -f https://raw.githubusercontent.com/jetstack/cert-manager/release-X.Y/deploy/manifests/00-crds.yaml
|
||||
kubectl delete -f https://github.com/jetstack/cert-manager/releases/download/vX.Y.Z/cert-manager.crds.yaml
|
||||
```
|
||||
|
||||
1. Install the CustomResourceDefinition resources separately
|
||||
|
||||
```plain
|
||||
kubectl apply -f https://raw.githubusercontent.com/jetstack/cert-manager/release-0.12/deploy/manifests/00-crds.yaml
|
||||
kubectl apply --validate=false -f https://github.com/jetstack/cert-manager/releases/download/vX.Y.Z/cert-manager.crds.yaml
|
||||
```
|
||||
|
||||
> **Note:**
|
||||
|
||||
@@ -75,6 +75,11 @@ of your Kubernetes cluster running Rancher server. You'll use the snapshot as a
|
||||
```plain
|
||||
helm fetch rancher-<CHART_REPO>/rancher
|
||||
```
|
||||
You can fetch the chart for the specific version you are upgrading to by adding in the `--version=` tag. For example:
|
||||
|
||||
```plain
|
||||
helm fetch rancher-<CHART_REPO>/rancher --version=v2.4.11
|
||||
```
|
||||
|
||||
### C. Upgrade Rancher
|
||||
|
||||
@@ -109,12 +114,14 @@ helm upgrade rancher rancher-<CHART_REPO>/rancher \
|
||||
|
||||
> **Note:** The above is an example, there may be more values from the previous step that need to be appended.
|
||||
|
||||
Alternatively, it's possible to reuse current values and make small changes with the `--reuse-values` flag. For example, to only change the Rancher version:
|
||||
Alternatively, it's possible to export the current values to a file and reference that file during upgrade. For example, to only change the Rancher version:
|
||||
|
||||
```
|
||||
helm get values rancher -n cattle-system -o yaml > values.yaml
|
||||
|
||||
helm upgrade rancher rancher-<CHART_REPO>/rancher \
|
||||
--namespace cattle-system \
|
||||
--reuse-values \
|
||||
-f values.yaml \
|
||||
--version=2.4.5
|
||||
```
|
||||
|
||||
|
||||
@@ -27,4 +27,4 @@ The workloads and services that you want to be controlled by Istio must meet [Is
|
||||
1. [Add deployments and services that have the Istio sidecar injected.]({{<baseurl>}}/rancher/v2.x/en/cluster-admin/tools/istio/setup/deploy-workloads)
|
||||
1. [Set up the Istio gateway. ]({{<baseurl>}}/rancher/v2.x/en/cluster-admin/tools/istio/setup/gateway)
|
||||
1. [Set up Istio's components for traffic management.]({{<baseurl>}}/rancher/v2.x/en/cluster-admin/tools/istio/setup/set-up-traffic-management)
|
||||
1. [Generate traffic and see Istio in action.](#generate-traffic-and-see-istio-in-action)
|
||||
1. [Generate traffic and see Istio in action.]({{<baseurl>}}/rancher/v2.x/en/istio/v2.5/setup/view-traffic/ )
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
---
|
||||
title: 4. Add Deployments and Services with the Istio Sidecar
|
||||
title: 3. Add Deployments and Services with the Istio Sidecar
|
||||
weight: 4
|
||||
aliases:
|
||||
- /rancher/v2.x/en/cluster-admin/tools/istio/setup/deploy-workloads
|
||||
@@ -18,13 +18,13 @@ Wait a few minutes for the workload to upgrade to have the istio sidecar. Click
|
||||
|
||||
There are a few ways to add new **Deployments** in your namespace
|
||||
|
||||
1. From the **Cluster Explorer** click on Workload > Overview
|
||||
1. Click **Create**
|
||||
1. Select **Deployment** from the various workload options
|
||||
1. Fill out the form, or **Edit as Yaml**
|
||||
1. Click **Create**
|
||||
1. From the **Cluster Explorer** click on **Workload > Overview.**
|
||||
1. Click **Create.**
|
||||
1. Select **Deployment** from the various workload options.
|
||||
1. Fill out the form, or **Edit as Yaml.**
|
||||
1. Click **Create.**
|
||||
|
||||
Alternatively, you can select the specific workload you want to deploy from worklod > specific workload and create from there.
|
||||
Alternatively, you can select the specific workload you want to deploy from the **Workload** section of the left navigation bar and create it from there.
|
||||
|
||||
To add a **Service** to your namespace
|
||||
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
---
|
||||
title: 5. Set up the Istio Gateway
|
||||
title: 4. Set up the Istio Gateway
|
||||
weight: 5
|
||||
aliases:
|
||||
- /rancher/v2.x/en/cluster-admin/tools/istio/setup/gateway
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
---
|
||||
title: 6. Set up Istio's Components for Traffic Management
|
||||
title: 5. Set up Istio's Components for Traffic Management
|
||||
weight: 6
|
||||
aliases:
|
||||
- /rancher/v2.x/en/cluster-admin/tools/istio/setup/set-up-traffic-management
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
---
|
||||
title: 7. Generate and View Traffic
|
||||
title: 6. Generate and View Traffic
|
||||
weight: 7
|
||||
aliases:
|
||||
- /rancher/v2.x/en/cluster-admin/tools/istio/setup/view-traffic
|
||||
@@ -10,7 +10,7 @@ This section describes how to view the traffic that is being managed by Istio.
|
||||
|
||||
# The Kiali Traffic Graph
|
||||
|
||||
The Istio overpage provides a link to the Kiali dashboard. From the Kiali dashboard, you are able to view graphs for each namespace. The Kiali graph provides a powerful way to visualize the topology of your Istio service mesh. It shows you which services communicate with each other.
|
||||
The Istio overview page provides a link to the Kiali dashboard. From the Kiali dashboard, you are able to view graphs for each namespace. The Kiali graph provides a powerful way to visualize the topology of your Istio service mesh. It shows you which services communicate with each other.
|
||||
|
||||
>**Prerequisite:** To enable traffic to show up in the graph, ensure you have prometheus installed in the cluster. Rancher-istio installs Kiali configured by default to work with the rancher-monitoring chart. You can use rancher-monitoring or install your own monitoring solution. Optional: you can change configuration on how data scraping occurs by setting the [Selectors & Scrape Configs]({{<baseurl>}}/rancher/v2.x/en/istio/setup/enable-istio-in-cluster/#selectors-scrape-configs) options.
|
||||
|
||||
@@ -19,7 +19,7 @@ To see the traffic graph,
|
||||
1. From the **Cluster Explorer**, select **Istio** from the nav dropdown.
|
||||
1. Click the **Kiali** link on the Istio **Overview** page.
|
||||
1. Click on **Graph** in the side nav.
|
||||
1. Change the namespace in the **Namesace** dropdown to view the traffic for each namespace.
|
||||
1. Change the namespace in the **Namespace** dropdown to view the traffic for each namespace.
|
||||
|
||||
If you refresh the URL to the BookInfo app several times, you should be able to see green arrows on the Kiali graph showing traffic to `v1` and `v3` of the `reviews` service. The control panel on the right side of the graph lets you configure details including how many minutes of the most recent traffic should be shown on the graph.
|
||||
|
||||
|
||||
@@ -4,7 +4,8 @@ description: Learn about the Docker registry and Kubernetes registry, their use
|
||||
weight: 3063
|
||||
aliases:
|
||||
- /rancher/v2.x/en/tasks/projects/add-registries/
|
||||
- /rancher/v2.x/en/k8s-in-rancher/registries
|
||||
- /rancher/v2.x/en/k8s-in-rancher/registries
|
||||
- /rancher/v2.x/en/k8s-resources/k8s-in-rancher/registries
|
||||
---
|
||||
Registries are Kubernetes secrets containing credentials used to authenticate with [private Docker registries](https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/).
|
||||
|
||||
|
||||
@@ -0,0 +1,181 @@
|
||||
---
|
||||
title: Migrating to Rancher v2.5 Logging
|
||||
weight: 5
|
||||
aliases:
|
||||
- /rancher/v2.x/en/logging/migrating
|
||||
---
|
||||
Starting in v2.5, the logging feature available within Rancher has been completely overhauled. The [logging operator](https://github.com/banzaicloud/logging-operator) from Banzai Cloud has been adopted; Rancher configures this tooling for use when deploying logging.
|
||||
|
||||
Among the many features and changes in the new logging functionality is the removal of project-specific logging configurations. Instead, one now configures logging at the namespace level. Cluster-level logging remains available, but configuration options differ.
|
||||
|
||||
> Note: The pre-v2.5 user interface is now referred to as the _Cluster Manager_. The v2.5+ dashboard is referred to as the _Cluster Explorer_.
|
||||
|
||||
## Installation
|
||||
|
||||
To install logging in Rancher v2.5+, refer to [installation instructions]({{<baseurl>}}/rancher/v2.x/en/logging/v2.5/#enabling-logging-for-rancher-managed-clusters).
|
||||
|
||||
## Terminology & Familiarity
|
||||
|
||||
In v2.5, logging configuration is centralized under a _Logging_ menu option available in the _Cluster Explorer_. It is from this menu option that logging for both cluster and namespace is configured.
|
||||
|
||||
> Note: Logging is installed on a per-cluster basis. You will need to navigate between clusters to configure logging for each cluster.
|
||||
|
||||
There are four key concepts to understand for v2.5+ logging:
|
||||
|
||||
1. Outputs
|
||||
|
||||
_Outputs_ are a configuration resource that determine a destination for collected logs. This is where settings for aggregators such as ElasticSearch, Kafka, etc. are stored. _Outputs_ are namespaced resources.
|
||||
|
||||
2. Flows
|
||||
|
||||
_Flows_ are a configuration resource that determine collection, filtering, and destination rules for logs. It is within a flow that one will configure what logs to collect, how to mutate or filter them, and which outputs to send the logs to. _Flows_ are namespaced resources, and can connect either to an _Output_ in the same namespace, or a _ClusterOutput_.
|
||||
|
||||
3. ClusterOutputs
|
||||
|
||||
_ClusterOutputs_ serve the same functionality as _Outputs_, except they are a cluster-scoped resource. _ClusterOutputs_ are necessary when collecting logs cluster-wide, or if you wish to provide an output to all namespaces in your cluster.
|
||||
|
||||
4. ClusterFlows
|
||||
|
||||
_ClusterFlows_ serve the same function as _Flows_, but at the cluster level. They are used to configure log collection for an entire cluster, instead of on a per-namespace level. _ClusterFlows_ are also where mutations and filters are defined, same as _Flows_ (in functionality).
|
||||
|
||||
# Cluster Logging
|
||||
|
||||
To configure cluster-wide logging for v2.5+ logging, one needs to setup a _ClusterFlow_. This object defines the source of logs, any transformations or filters to be applied, and finally the output(s) for the logs.
|
||||
|
||||
> Important: _ClusterFlows_ must be defined within the `cattle-logging-system` namespace. _ClusterFlows_ will not work if defined in any other namespace.
|
||||
|
||||
In legacy logging, in order to collect logs from across the entire cluster, one only needed to enable cluster-level logging and define the desired output. This basic approach remains in v2.5+ logging. To replicate legacy cluster-level logging, follow these steps:
|
||||
|
||||
1. Define a _ClusterOutput_ according to the instructions found under [Output Configuration](#output-configuration)
|
||||
2. Create a _ClusterFlow_, ensuring that it is set to be created in the `cattle-logging-system` namespace
|
||||
1. Remove all _Include_ and _Exclude_ rules from the flow definition. This ensures that all logs are gathered.
|
||||
2. You do not need to configure any filters if you do not wish - default behavior does not require their creation
|
||||
3. Define your cluster output(s)
|
||||
|
||||
This will result in logs from all sources in the cluster (all pods, and all system components) being collected and sent to the output(s) you defined in the _ClusterFlow_.
|
||||
|
||||
# Project Logging
|
||||
|
||||
Logging in v2.5+ is not project-aware. This means that in order to collect logs from pods running in project namespaces, you will need to define _Flows_ for those namespaces.
|
||||
|
||||
To collect logs from a specific namespace, follow these steps:
|
||||
|
||||
1. Define an _Output_ or _ClusterOutput_ according to the instructions found under [Output Configuration](#output-configuration)
|
||||
2. Create a _Flow_, ensuring that it is set to be created in the namespace in which you want to gather logs.
|
||||
1. If you wish to define _Include_ or _Exclude_ rules, you may do so. Otherwise, removal of all rules will result in all pods in the target namespace having their logs collected.
|
||||
2. You do not need to configure any filters if you do not wish - default behavior does not require their creation
|
||||
3. Define your output(s) - these can be either _ClusterOutput_ or _Output_ objects.
|
||||
|
||||
This will result in logs from all sources in the namespace (pods) being collected and sent to the output(s) you defined in your _Flow_.
|
||||
|
||||
> To collect logs from a project, repeat the above steps for every namespace within the project. Alternatively, you can label your project workloads with a common label (e.g. `project=my-project`) and use a _ClusterFlow_ to collect logs from all pods matching this label.
|
||||
|
||||
# Output Configuration
|
||||
In legacy logging, there are five logging destinations to choose from: Elasticsearch, Splunk, Kafka, Fluentd, and Syslog. With the exception of Syslog, all of these destinations are available in logging v2.5+.
|
||||
|
||||
|
||||
## Elasticsearch
|
||||
|
||||
| Legacy Logging | v2.5+ Logging | Notes |
|
||||
|-----------------------------------------------|-----------------------------------|-----------------------------------------------------------|
|
||||
| Endpoint | Target -> Host | Make sure to specify Scheme (https/http), as well as Port |
|
||||
| X-Pack Security -> Username | Access -> User | |
|
||||
| X-Pack Security -> Password | Access -> Password | Password must now be stored in a secret |
|
||||
| SSL Configuration -> Client Private Key | SSL -> Client Key | Key must now be stored in a secret |
|
||||
| SSL Configuration -> Client Certificate | SSL -> Client Cert | Certificate must now be stored in a secret |
|
||||
| SSL Configuration -> Client Key Password | SSL -> Client Key Pass | Password must now be stored in a secret |
|
||||
| SSL Configuration -> Enabled SSL Verification | SSL -> Certificate Authority File | Certificate must now be stored in a secret |
|
||||
|
||||
|
||||
In legacy logging, indices were automatically created according to the format in the "Index Patterns" section. In v2.5 logging, default behavior has been changed to logging to a single index. You can still configure index pattern functionality on the output object by editing as YAML and inputting the following values:
|
||||
|
||||
```
|
||||
...
|
||||
spec:
|
||||
elasticsearch:
|
||||
...
|
||||
logstash_format: true
|
||||
logstash_prefix: <desired prefix>
|
||||
logstash_dateformat: "%Y-%m-%d"
|
||||
```
|
||||
|
||||
Replace `<desired prefix>` with the prefix for the indices that will be created. In legacy logging, this defaulted to the name of the cluster.
|
||||
|
||||
## Splunk
|
||||
|
||||
| Legacy Logging | v2.5+ Logging | Notes |
|
||||
|------------------------------------------|----------------------------------------|----------------------------------------------------------------------------------------|
|
||||
| HEC Configuration -> Endpoint | Target -> Host | Protocol (https/http) and port must be defined separately from the host |
|
||||
| HEC Configuration -> Token | Access -> Token | Token must now be stored as a secret |
|
||||
| HEC Configuration -> Index | Edit as YAML -> `index` | `index` field must be added as YAML key under `spec.splunkHec` |
|
||||
| HEC Configuration -> Source | Edit as YAML -> `source` | `source` field must be added as YAML key under `spec.splunkHec` |
|
||||
| SSL Configuration -> Client Private Key | Edit as YAML -> `client_key` | `client_key` field must be added as YAML key under `spec.splunkHec`. See (1) |
|
||||
| SSL Configuration -> Client Certificate | Edit as YAML -> `client_cert` | `client_cert` field must be added as YAML key under `spec.splunkHec`. See (1) |
|
||||
| SSL Configuration -> Client Key Password | _Not Supported_ | Specifying a password for the client private key is not currently supported. |
|
||||
| SSL Configuration -> SSL Verify | Edit as YAML -> `ca_file` or `ca_path` | `ca_file` or `ca_path` field must be added as YAML key under `spec.splunkHec`. See (2) |
|
||||
|
||||
_(1) `client_key` and `client_cert` values must be paths to the key and cert files, respectively. These files must be mounted into the `rancher-logging-fluentd` pod in order to be used._
|
||||
|
||||
_(2) Users can configure either `ca_file` (a path to a PEM-encoded CA certificate) or `ca_path` (a path to a directory containing CA certificates in PEM format). These files must be mounted into the `rancher-logging-fluentd` pod in order to be used._
|
||||
|
||||
## Kafka
|
||||
|
||||
| Legacy Logging | v2.5+ Logging | Notes |
|
||||
|-----------------------------------------|----------------------------|------------------------------------------------------|
|
||||
| Kafka Configuration -> Endpoint Type | - | Zookeeper is no longer supported as an endpoint type |
|
||||
| Kafka Configuration -> Endpoint | Target -> Brokers | Comma-separated list of brokers (host:port) |
|
||||
| Kafka Configuration -> Topic | Target -> Default Topic | |
|
||||
| SSL Configuration -> Client Private Key | SSL -> SSL Client Cert | Certificate must be stored as a secret |
|
||||
| SSL Configuration -> Client Certificate | SSL -> SSL Client Cert Key | Key must be stored as a secret |
|
||||
| SSL Configuration -> CA Certificate PEM | SSL -> SSL CA Cert | Certificate must be stored as a secret |
|
||||
| SASL Configuration -> Username | Access -> Username | Username must be stored in a secret |
|
||||
| SASL Configuration -> Password | Access -> Password | Password must be stored in a secret |
|
||||
| SASL Configuration -> Scram Mechanism | Access -> Scram Mechanism | Input mechanism as string, e.g. "sha256" or "sha512" |
|
||||
|
||||
## Fluentd
|
||||
|
||||
As of v2.5.2, it is only possible to add a single Fluentd server using the "Edit as Form" option. To add multiple servers, edit the output as YAML and input multiple servers.
|
||||
|
||||
| Legacy Logging | v2.5+ Logging | Notes |
|
||||
|------------------------------------------|-----------------------------------------------------|----------------------------------------------------------------------|
|
||||
| Fluentd Configuration -> Endpoint | Target -> Host, Port | Input the host and port separately |
|
||||
| Fluentd Configuration -> Shared Key | Access -> Shared Key | Shared key must be stored as a secret |
|
||||
| Fluentd Configuration -> Username | Access -> Username | Username must be stored as a secret |
|
||||
| Fluentd Configuration -> Password | Access -> Password | Password must be stored as a secret |
|
||||
| Fluentd Configuration -> Hostname | Edit as YAML -> `host` | `host` field set as YAML key under `spec.forward.servers[n]` |
|
||||
| Fluentd Configuration -> Weight | Edit as YAML -> `weight` | `weight` field set as YAML key under `spec.forward.servers[n]` |
|
||||
| SSL Configuration -> Use TLS | - | Do not need to explicitly enable. Define client cert fields instead. |
|
||||
| SSL Configuration -> Client Private Key | Edit as YAML -> `tls_private_key_path` | Field set as YAML key under `spec.forward`. See (1) |
|
||||
| SSL Configuration -> Client Certificate | Edit as YAML -> `tls_client_cert_path` | Field set as YAML key under `spec.forward`. See (1) |
|
||||
| SSL Configuration -> Client Key Password | Edit as YAML -> `tls_client_private_key_passphrase` | Field set as YAML key under `spec.forward`. See (1) |
|
||||
| SSL Configuration -> SSL Verify | Edit as YAML -> `tls_insecure_mode` | Field set as YAML key under `spec.forward`. Default: `false` |
|
||||
| SSL Configuration -> CA Certificate PEM | Edit as YAML -> `tls_cert_path` | Field set as YAML key under `spec.forward`. See (1) |
|
||||
| Enable Gzip Compression | - | No longer supported in v2.5+ logging |
|
||||
|
||||
_(1) These values are to be specified as paths to files. Those files must be mounted into the `rancher-logging-fluentd` pod in order to be used._
|
||||
|
||||
## Syslog
|
||||
|
||||
As of v2.5.2, syslog is not currently supported as an output using v2.5+ logging.
|
||||
|
||||
## Custom Log Fields
|
||||
|
||||
In order to add custom log fields, you will need to add the following YAML to your flow configuration:
|
||||
|
||||
```
|
||||
...
|
||||
spec:
|
||||
filters:
|
||||
- record_modifier:
|
||||
records:
|
||||
- foo: "bar"
|
||||
```
|
||||
|
||||
(replace `foo: "bar"` with custom log fields you wish to add)
|
||||
|
||||
# System Logging
|
||||
|
||||
In legacy logging, collecting logs from system components was accomplished by checking a box labeled "Include System Log" when setting up cluster logging. In v2.5+ logging, system logs are gathered in one of two ways:
|
||||
|
||||
1. Gather all cluster logs, not specifying any match or exclusion rules. This results in all container logs from the cluster being collected, which includes system logs.
|
||||
2. Specifically target system logs by adding match rules for system components. Specific match rules depend on the component being collected.
|
||||
@@ -90,7 +90,7 @@ This alert type monitor for events that affect one of the Kubernetes master comp
|
||||
|
||||
- **Group Wait Time**: How long to wait to buffer alerts of the same group before sending initially, default to 30 seconds.
|
||||
- **Group Interval Time**: How long to wait before sending an alert that has been added to a group which contains already fired alerts, default to 30 seconds.
|
||||
- **Repeat Wait Time**: How long to wait before sending an alert that has been added to a group which contains already fired alerts, default to 1 hour.
|
||||
- **Repeat Wait Time**: How long to wait before re-sending a given alert that has already been sent, default to 1 hour.
|
||||
|
||||
{{% /accordion %}}
|
||||
{{% accordion id="resource-event" label="Resource Event Alerts" %}}
|
||||
@@ -125,7 +125,7 @@ This alert type monitors for specific events that are thrown from a resource typ
|
||||
|
||||
- **Group Wait Time**: How long to wait to buffer alerts of the same group before sending initially, default to 30 seconds.
|
||||
- **Group Interval Time**: How long to wait before sending an alert that has been added to a group which contains already fired alerts, default to 30 seconds.
|
||||
- **Repeat Wait Time**: How long to wait before sending an alert that has been added to a group which contains already fired alerts, default to 1 hour.
|
||||
- **Repeat Wait Time**: How long to wait before re-sending a given alert that has already been sent, default to 1 hour.
|
||||
|
||||
{{% /accordion %}}
|
||||
{{% accordion id="node" label="Node Alerts" %}}
|
||||
@@ -152,7 +152,7 @@ This alert type monitors for events that occur on a specific node.
|
||||
|
||||
- **Group Wait Time**: How long to wait to buffer alerts of the same group before sending initially, default to 30 seconds.
|
||||
- **Group Interval Time**: How long to wait before sending an alert that has been added to a group which contains already fired alerts, default to 30 seconds.
|
||||
- **Repeat Wait Time**: How long to wait before sending an alert that has been added to a group which contains already fired alerts, default to 1 hour.
|
||||
- **Repeat Wait Time**: How long to wait before re-sending a given alert that has already been sent, default to 1 hour.
|
||||
|
||||
{{% /accordion %}}
|
||||
{{% accordion id="node-selector" label="Node Selector Alerts" %}}
|
||||
@@ -179,7 +179,7 @@ This alert type monitors for events that occur on any node on marked with a labe
|
||||
|
||||
- **Group Wait Time**: How long to wait to buffer alerts of the same group before sending initially, default to 30 seconds.
|
||||
- **Group Interval Time**: How long to wait before sending an alert that has been added to a group which contains already fired alerts, default to 30 seconds.
|
||||
- **Repeat Wait Time**: How long to wait before sending an alert that has been added to a group which contains already fired alerts, default to 1 hour.
|
||||
- **Repeat Wait Time**: How long to wait before re-sending a given alert that has already been sent, default to 1 hour.
|
||||
|
||||
{{% /accordion %}}
|
||||
{{% accordion id="cluster-expression" label="Metric Expression Alerts" %}}
|
||||
@@ -224,7 +224,7 @@ This alert type monitors for the overload from Prometheus expression querying, i
|
||||
|
||||
- **Group Wait Time**: How long to wait to buffer alerts of the same group before sending initially, default to 30 seconds.
|
||||
- **Group Interval Time**: How long to wait before sending an alert that has been added to a group which contains already fired alerts, default to 30 seconds.
|
||||
- **Repeat Wait Time**: How long to wait before sending an alert that has been added to a group which contains already fired alerts, default to 1 hour.
|
||||
- **Repeat Wait Time**: How long to wait before re-sending a given alert that has already been sent, default to 1 hour.
|
||||
|
||||
{{% /accordion %}}
|
||||
|
||||
|
||||
+1
@@ -4,6 +4,7 @@ weight: 5
|
||||
aliases:
|
||||
- /rancher/v2.x/en/project-admin/tools/monitoring/custom-metrics
|
||||
- /rancher/v2.x/en/monitoring-alerting/legacy/monitoring/cluster-monitoring/custom-metrics
|
||||
- /rancher/v2.x/en/cluster-admin/tools/monitoring/custom-metrics/
|
||||
---
|
||||
|
||||
After you've enabled [cluster level monitoring]({{< baseurl >}}/rancher/v2.x/en/monitoring-alerting/legacy/monitoring/cluster-monitoring/#enabling-cluster-monitoring), You can view the metrics data from Rancher. You can also deploy the Prometheus custom metrics adapter then you can use the HPA with metrics stored in cluster monitoring.
|
||||
|
||||
@@ -105,6 +105,8 @@ Rancher allows any users who are authenticated by Kubernetes and have access the
|
||||
|
||||
However, users can choose to log in to Grafana as an [Admin](https://grafana.com/docs/grafana/latest/permissions/organization_roles/#admin-role) if necessary. The default Admin username and password for the Grafana instance will be `admin`/`prom-operator`, but alternative credentials can also be supplied on deploying or upgrading the chart.
|
||||
|
||||
> **Persistent Dashboards:** To allow the Grafana dashboard to persist after it restarts, add the dashboard configuration JSON into a ConfigMap. ConfigMaps also allow the dashboards to be deployed with a GitOps or CD based approach. This allows the dashboard to be put under version control. For details, refer to [this section.](./persist-grafana)
|
||||
|
||||
To see the Grafana UI, install `rancher-monitoring`. Then go to the **Cluster Explorer.** In the top left corner, click **Cluster Explorer > Monitoring.** Then click **Grafana.
|
||||
|
||||
<figcaption>Cluster Compute Resources Dashboard in Grafana</figcaption>
|
||||
@@ -113,8 +115,6 @@ To see the Grafana UI, install `rancher-monitoring`. Then go to the **Cluster Ex
|
||||
<figcaption>Default Dashboards in Grafana</figcaption>
|
||||
|
||||
|
||||
To allow the Grafana dashboard to persist after it restarts, you will need to add the configuration JSON into a ConfigMap. You can add this configuration to the ConfigMap using the Rancher UI.
|
||||
|
||||
### Prometheus UI
|
||||
|
||||
To see the Prometheus UI, install `rancher-monitoring`. Then go to the **Cluster Explorer.** In the top left corner, click **Cluster Explorer > Monitoring.** Then click **Prometheus Graph.**
|
||||
@@ -183,4 +183,4 @@ At least 50Gi storage is recommended.
|
||||
|
||||
# Known Issues
|
||||
|
||||
There is a [known issue](https://github.com/rancher/rancher/issues/28787#issuecomment-693611821) that K3s clusters require more default memory. If you are enabling monitoring on a K3s cluster, we recommend to setting `prometheus.prometheusSpec.resources.memory.limit` to 2500Mi` and `prometheus.prometheusSpec.resources.memory.request` to 1750Mi.
|
||||
There is a [known issue](https://github.com/rancher/rancher/issues/28787#issuecomment-693611821) that K3s clusters require more default memory. If you are enabling monitoring on a K3s cluster, we recommend to setting `prometheus.prometheusSpec.resources.memory.limit` to 2500 Mi and `prometheus.prometheusSpec.resources.memory.request` to 1750 Mi.
|
||||
|
||||
@@ -5,7 +5,7 @@ aliases:
|
||||
- /rancher/v2.x/en/monitoring-alerting/migrating
|
||||
---
|
||||
|
||||
If you previously enabled Monitoring, Alerting, or Notifiers in Rancher prior to v2.5, there is no upgrade path for switching to the new monitoring/alerting solution. You will need to disable monitoring/alerting/notifiers in the same way it was disabled in Rancher v2.4 before deploying the new monitoring solution via Cluster Explorer.
|
||||
If you previously enabled Monitoring, Alerting, or Notifiers in Rancher prior to v2.5, there is no automatic upgrade path for switching to the new monitoring/alerting solution. Before deploying the new monitoring solution via Cluster Explore, you will need to disable and remove all existing custom alerts, notifiers and monitoring installations for the whole cluster and in all projects.
|
||||
|
||||
### Monitoring Prior to Rancher v2.5
|
||||
|
||||
@@ -32,3 +32,86 @@ For more information on how to configure Monitoring & Alerting V2, see [this pag
|
||||
Project owners and members no longer get access to Grafana or Prometheus by default. If view-only users had access to Grafana, they would be able to see data from any namespace. For Kiali, any user can edit things they don’t own in any namespace.
|
||||
|
||||
For more information about role-based access control in `rancher-monitoring`, refer to [this page.](../rbac)
|
||||
|
||||
### Migrating from Monitoring V1 to Monitoring V2
|
||||
|
||||
While there is no automatic migration available, it is possible to manually migrate custom Grafana dashboards and alerts that were created in Monitoring V1 to Monitoring V2.
|
||||
|
||||
Before you can install Monitoring V2, Monitoring V1 needs to be uninstalled completely. In order to uninstall Monitoring V1:
|
||||
|
||||
* Remove all cluster and project specific alerts and alerts groups
|
||||
* Remove all notifiers
|
||||
* Disable all project monitoring installations under Cluster -> Project -> Tools -> Monitoring
|
||||
* Ensure that all project-monitoring apps in all projects have been removed and are not recreated after a few minutes
|
||||
* Disable the cluster monitoring installation under Cluster -> Tools -> Monitoring
|
||||
* Ensure that the cluster-monitoring app and the monitoring-operator app in the System project have been removed and are not recreated after a few minutes
|
||||
|
||||
#### Migrating Grafana Dashboards
|
||||
|
||||
You can migrate any dashboard added to Grafana in Monitoring V1 to Monitoring V2. In Monitoring V1 you can export an existing dashboard like this:
|
||||
|
||||
* Sign into Grafana
|
||||
* Navigate to the dashboard you want to export
|
||||
* Go to the dashboard settings
|
||||
* Copy the [JSON Model](https://grafana.com/docs/grafana/latest/dashboards/json-model/)
|
||||
|
||||
In the JSON Model, change all `datasource` fields from `RANCHER_MONITORING` to `Prometheus`. You can easily do this by replacing all occurrences of `"datasource": "RANCHER_MONITORING"` with `"datasource": "Prometheus"`.
|
||||
|
||||
If Grafana is backed by a persistent volume, you can now [import](https://grafana.com/docs/grafana/latest/dashboards/export-import/) this JSON Model into the Monitoring V2 Grafana UI.
|
||||
It is recommended to provide the dashboard to Grafana with a ConfigMap in the `cattle-dashboards` namespace that has the label `grafana_dashboard: "1"`:
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: custom-dashboard
|
||||
namespace: cattle-dashboards
|
||||
labels:
|
||||
grafana_dashboard: "1"
|
||||
data:
|
||||
custom-dashboard.json: |
|
||||
{
|
||||
...
|
||||
}
|
||||
```
|
||||
|
||||
Once this ConfigMap is created, the dashboard will automatically be added to Grafana.
|
||||
|
||||
#### Migrating Alerts
|
||||
|
||||
It is only possible to directly migrate expression-based alerts to Monitoring V2. Fortunately, the event-based alerts that could be set up to alert on system component, node or workload events, are already covered out-of-the-box by the alerts that are part of Monitoring V2. So it is not necessary to migrate them.
|
||||
|
||||
To migrate the following expression alert
|
||||
|
||||
{{< img "/img/rancher/monitoring/migration/alert_2.4_to_2.5_source.png" "">}}
|
||||
|
||||
you have to either create a PrometheusRule configuration like this in any namespace
|
||||
|
||||
```yaml
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: PrometheusRule
|
||||
metadata:
|
||||
name: custom-rules
|
||||
namespace: default
|
||||
spec:
|
||||
groups:
|
||||
- name: custom.rules
|
||||
rules:
|
||||
- alert: Custom Expression Alert
|
||||
expr: prometheus_query > 5
|
||||
for: 5m
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: "The result of prometheus_query has been larger than 5 for 5m. Current value {{ $value }}"
|
||||
```
|
||||
|
||||
or add the Prometheus Rule through the Cluster Explorer
|
||||
|
||||
{{< img "/img/rancher/monitoring/migration/alert_2.4_to_2.5_target.png" "">}}
|
||||
|
||||
For more details on how to configure PrometheusRules in Monitoring V2 see [Monitoring Configuration]({{<baseurl>}}/rancher/v2.x/en/monitoring-alerting/v2.5/configuration#prometheusrules).
|
||||
|
||||
#### Migrating notifiers
|
||||
|
||||
There is no direct equivalent for how notifiers work in Monitoring V1. Instead you have to replicate the desired setup with [Routes and Receivers]({{<baseurl>}}/rancher/v2.x/en/monitoring-alerting/v2.5/configuration#alertmanager-config) in Monitoring V2.
|
||||
|
||||
@@ -0,0 +1,35 @@
|
||||
---
|
||||
title: Persistent Grafana Dashboards
|
||||
weight: 4
|
||||
---
|
||||
|
||||
To allow the Grafana dashboard to persist after the Grafana instance restarts, add the dashboard configuration JSON into a ConfigMap. ConfigMaps also allow the dashboards to be deployed with a GitOps or CD based approach. This allows the dashboard to be put under version control.
|
||||
|
||||
> **Prerequisites:**
|
||||
>
|
||||
> - The monitoring application needs to be installed.
|
||||
> - You must have the cluster-admin ClusterRole permission.
|
||||
|
||||
1. Open the Grafana dashboard. From the **Cluster Explorer,** click **Cluster Explorer > Monitoring.**
|
||||
1. Log in to Grafana. Note: The default Admin username and password for the Grafana instance is `admin/prom-operator`. (Regardless of who has the password, cluster administrator permission in Rancher is still required access the Grafana instance.) Alternative credentials can also be supplied on deploying or upgrading the chart.
|
||||
1. Go to the dashboard that you want to persist. In the top navigation menu, go to the dashboard settings by clicking the gear icon.
|
||||
1. In the left navigation menu, click **JSON Model.**
|
||||
1. Copy the JSON data structure that appears.
|
||||
1. Create a ConfigMap in the `cattle-dashboards` namespace. The ConfigMap needs to have the label `grafana_dashboard: "1"`. Paste the JSON into the ConfigMap in the format shown in the example below:
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
labels:
|
||||
grafana_dashboard: "1"
|
||||
name: <dashboard-name>
|
||||
namespace: cattle-dashboards
|
||||
data:
|
||||
<dashboard-name>.json: |-
|
||||
<copied-json>
|
||||
```
|
||||
|
||||
**Result:** After the ConfigMap is created, it should show up on the Grafana UI and be persisted even if the Grafana pod is restarted.
|
||||
|
||||
Dashboards that are persisted using ConfigMaps cannot be deleted from the Grafana UI. If you attempt to delete the dashboard in the Grafana UI, you will see the error message "Dashboard cannot be deleted because it was provisioned." To delete the dashboard, you will need to delete the ConfigMap.
|
||||
@@ -49,7 +49,7 @@ Only those with the the cluster-admin / admin / edit `ClusterRole` should be abl
|
||||
Only those with who have some k8s `ClusterRole` should be able to:
|
||||
|
||||
- View the configuration of Prometheuses that are deployed within the cluster
|
||||
- View the configuraiton of Alertmanagers that are deployed within the cluster
|
||||
- View the configuration of Alertmanagers that are deployed within the cluster
|
||||
- View the scrape configuration of Prometheus deployments via ServiceMonitor and PodMonitor CRs
|
||||
- View the alerting / recording rules of a Prometheus deployment via PrometheusRules CRs
|
||||
|
||||
@@ -96,5 +96,5 @@ If cluster-admins would like to provide additional admin/edit access to users ou
|
||||
|
||||
| k8s Resources | Namespace | Can it cause impact outside of a namespace / project? | Impact |
|
||||
|----------------------------| ------| ------| ----------------------------|
|
||||
| <ul><li>`secrets`</li><li>`configmaps`</li></ul>| `cattle-monitoring-system` | Yes, Configs and Secrets in this namespace can impact the entire monitoring / alerting pipeline. | User will be able to create or edit Secrets / ConfigMaps such as the Alertmanager Config, Prometheus Adapter Config, TLS secrets, additional Grafana datasoruces, etc. This can have broad impact on all cluster monitoring / alerting. |
|
||||
| <ul><li>`secrets`</li><li>`configmaps`</li></ul>| `cattle-monitoring-system` | Yes, Configs and Secrets in this namespace can impact the entire monitoring / alerting pipeline. | User will be able to create or edit Secrets / ConfigMaps such as the Alertmanager Config, Prometheus Adapter Config, TLS secrets, additional Grafana datasources, etc. This can have broad impact on all cluster monitoring / alerting. |
|
||||
| <ul><li>`secrets`</li><li>`configmaps`</li></ul>| `cattle-dashboards` | Yes, Configs and Secrets in this namespace can create dashboards that make queries on all metrics collected at a cluster-level. | User will be able to create Secrets / ConfigMaps that persist new Grafana Dashboards only. |
|
||||
|
||||
+28
-29
@@ -1,21 +1,21 @@
|
||||
---
|
||||
title: CIS Benchmark Rancher Self-Assessment Guide - v2.4
|
||||
title: CIS 1.5 Benchmark - Self-Assessment Guide - Rancher v2.5
|
||||
weight: 204
|
||||
---
|
||||
|
||||
### CIS Kubernetes Benchmark v1.5 - Rancher v2.4 with Kubernetes v1.15
|
||||
### CIS v1.5 Kubernetes Benchmark - Rancher v2.5 with Kubernetes v1.15
|
||||
|
||||
[Click here to download a PDF version of this document](https://releases.rancher.com/documents/security/2.4/Rancher_Benchmark_Assessment.pdf)
|
||||
[Click here to download a PDF version of this document](https://releases.rancher.com/documents/security/2.5/Rancher_1.5_Benchmark_Assessment.pdf)
|
||||
|
||||
#### Overview
|
||||
|
||||
This document is a companion to the Rancher v2.4 security hardening guide. The hardening guide provides prescriptive guidance for hardening a production installation of Rancher, and this benchmark guide is meant to help you evaluate the level of security of the hardened cluster against each control in the benchmark.
|
||||
This document is a companion to the Rancher v2.5 security hardening guide. The hardening guide provides prescriptive guidance for hardening a production installation of Rancher, and this benchmark guide is meant to help you evaluate the level of security of the hardened cluster against each control in the benchmark.
|
||||
|
||||
This guide corresponds to specific versions of the hardening guide, Rancher, Kubernetes, and the CIS Benchmark:
|
||||
This guide corresponds to specific versions of the hardening guide, Rancher, CIS Benchmark, and Kubernetes:
|
||||
|
||||
Self Assessment Guide Version | Rancher Version | Hardening Guide Version | Kubernetes Version | CIS Benchmark Version
|
||||
---------------------------|----------|---------|-------|-----
|
||||
Self Assessment Guide v2.4 | Rancher v2.4 | Hardening Guide v2.4 | Kubernetes v1.15 | Benchmark v1.5
|
||||
Hardening Guide Version | Rancher Version | CIS Benchmark Version | Kubernetes Version
|
||||
---------------------------|----------|---------|-------
|
||||
Hardening Guide with CIS 1.5 Benchmark | Rancher v2.5 | CIS v1.5| Kubernetes v1.15
|
||||
|
||||
Because Rancher and RKE install Kubernetes services as Docker containers, many of the control verification checks in the CIS Kubernetes Benchmark don't apply and will have a result of `Not Applicable`. This guide will walk through the various controls and provide updated example commands to audit compliance in Rancher-created clusters.
|
||||
|
||||
@@ -36,7 +36,7 @@ When performing the tests, you will need access to the Docker command line on th
|
||||
|
||||
---
|
||||
## 1 Master Node Security Configuration
|
||||
### 1.1 Master Node Configuration Files
|
||||
### 1.1 Master Node Configuration Files
|
||||
|
||||
#### 1.1.1 Ensure that the API server pod specification file permissions are set to `644` or more restrictive (Scored)
|
||||
|
||||
@@ -152,7 +152,7 @@ Run the below command (based on the etcd data directory found above).
|
||||
For example,
|
||||
``` bash
|
||||
chown etcd:etcd /var/lib/etcd
|
||||
```
|
||||
```
|
||||
|
||||
**Audit Script:** 1.1.12.sh
|
||||
|
||||
@@ -186,7 +186,7 @@ docker inspect etcd | jq -r '.[].HostConfig.Binds[]' | grep "${test_dir}" | cut
|
||||
RKE does not store the kubernetes default kubeconfig credentials file on the nodes. It’s presented to user where RKE is run.
|
||||
We recommend that this `kube_config_cluster.yml` file be kept in secure store.
|
||||
|
||||
#### 1.1.14 Ensure that the admin.conf file ownership is set to `root:root` (Scored)
|
||||
#### 1.1.14 Ensure that the admin.conf file ownership is set to `root:root` (Scored)
|
||||
|
||||
**Result:** Not Applicable
|
||||
|
||||
@@ -246,7 +246,7 @@ stat -c %U:%G /etc/kubernetes/ssl
|
||||
'root:root' is present
|
||||
```
|
||||
|
||||
#### 1.1.20 Ensure that the Kubernetes PKI certificate file permissions are set to `644` or more restrictive (Scored)
|
||||
#### 1.1.20 Ensure that the Kubernetes PKI certificate file permissions are set to `644` or more restrictive (Scored)
|
||||
|
||||
**Result:** PASS
|
||||
|
||||
@@ -729,7 +729,7 @@ on the master node and set the below parameter.
|
||||
'0' is equal to '0'
|
||||
```
|
||||
|
||||
#### 1.2.20 Ensure that the `--secure-port` argument is not set to `0` (Scored)
|
||||
#### 1.2.20 Ensure that the `--secure-port` argument is not set to `0` (Scored)
|
||||
|
||||
**Result:** PASS
|
||||
|
||||
@@ -950,7 +950,7 @@ to the public key file for service accounts:
|
||||
'--service-account-key-file' is present
|
||||
```
|
||||
|
||||
#### 1.2.29 Ensure that the `--etcd-certfile` and `--etcd-keyfile` arguments are set as appropriate (Scored)
|
||||
#### 1.2.29 Ensure that the `--etcd-certfile` and `--etcd-keyfile` arguments are set as appropriate (Scored)
|
||||
|
||||
**Result:** PASS
|
||||
|
||||
@@ -1308,7 +1308,7 @@ on the master node and set the below parameter.
|
||||
'false' is equal to 'false'
|
||||
```
|
||||
|
||||
#### 1.4.2 Ensure that the `--bind-address` argument is set to `127.0.0.1` (Scored)
|
||||
#### 1.4.2 Ensure that the `--bind-address` argument is set to `127.0.0.1` (Scored)
|
||||
|
||||
**Result:** PASS
|
||||
|
||||
@@ -1482,7 +1482,7 @@ node and either remove the `--peer-auto-tls` parameter or set it to `false`.
|
||||
## 3 Control Plane Configuration
|
||||
### 3.2 Logging
|
||||
|
||||
#### 3.2.1 Ensure that a minimal audit policy is created (Scored)
|
||||
#### 3.2.1 Ensure that a minimal audit policy is created (Scored)
|
||||
|
||||
**Result:** PASS
|
||||
|
||||
@@ -1543,7 +1543,7 @@ chmod 644 /etc/kubernetes/ssl/kubecfg-kube-proxy.yaml
|
||||
**Audit:**
|
||||
|
||||
```
|
||||
/bin/sh -c 'if test -e /etc/kubernetes/ssl/kubecfg-kube-proxy.yaml; then stat -c %a /etc/kubernetes/ssl/kubecfg-kube-proxy.yaml; fi'
|
||||
/bin/sh -c 'if test -e /etc/kubernetes/ssl/kubecfg-kube-proxy.yaml; then stat -c %a /etc/kubernetes/ssl/kubecfg-kube-proxy.yaml; fi'
|
||||
```
|
||||
|
||||
**Expected result**:
|
||||
@@ -1558,7 +1558,7 @@ chmod 644 /etc/kubernetes/ssl/kubecfg-kube-proxy.yaml
|
||||
|
||||
**Remediation:**
|
||||
Run the below command (based on the file location on your system) on the each worker node.
|
||||
For example,
|
||||
For example,
|
||||
|
||||
``` bash
|
||||
chown root:root /etc/kubernetes/ssl/kubecfg-kube-proxy.yaml
|
||||
@@ -1567,7 +1567,7 @@ chown root:root /etc/kubernetes/ssl/kubecfg-kube-proxy.yaml
|
||||
**Audit:**
|
||||
|
||||
```
|
||||
/bin/sh -c 'if test -e /etc/kubernetes/ssl/kubecfg-kube-proxy.yaml; then stat -c %U:%G /etc/kubernetes/ssl/kubecfg-kube-proxy.yaml; fi'
|
||||
/bin/sh -c 'if test -e /etc/kubernetes/ssl/kubecfg-kube-proxy.yaml; then stat -c %U:%G /etc/kubernetes/ssl/kubecfg-kube-proxy.yaml; fi'
|
||||
```
|
||||
|
||||
**Expected result**:
|
||||
@@ -1591,7 +1591,7 @@ chmod 644 /etc/kubernetes/ssl/kubecfg-kube-node.yaml
|
||||
**Audit:**
|
||||
|
||||
```
|
||||
/bin/sh -c 'if test -e /etc/kubernetes/ssl/kubecfg-kube-node.yaml; then stat -c %a /etc/kubernetes/ssl/kubecfg-kube-node.yaml; fi'
|
||||
/bin/sh -c 'if test -e /etc/kubernetes/ssl/kubecfg-kube-node.yaml; then stat -c %a /etc/kubernetes/ssl/kubecfg-kube-node.yaml; fi'
|
||||
```
|
||||
|
||||
**Expected result**:
|
||||
@@ -1615,7 +1615,7 @@ chown root:root /etc/kubernetes/ssl/kubecfg-kube-node.yaml
|
||||
**Audit:**
|
||||
|
||||
```
|
||||
/bin/sh -c 'if test -e /etc/kubernetes/ssl/kubecfg-kube-node.yaml; then stat -c %U:%G /etc/kubernetes/ssl/kubecfg-kube-node.yaml; fi'
|
||||
/bin/sh -c 'if test -e /etc/kubernetes/ssl/kubecfg-kube-node.yaml; then stat -c %U:%G /etc/kubernetes/ssl/kubecfg-kube-node.yaml; fi'
|
||||
```
|
||||
|
||||
**Expected result**:
|
||||
@@ -1661,7 +1661,7 @@ chown root:root <filename>
|
||||
**Audit:**
|
||||
|
||||
```
|
||||
/bin/sh -c 'if test -e /etc/kubernetes/ssl/kube-ca.pem; then stat -c %U:%G /etc/kubernetes/ssl/kube-ca.pem; fi'
|
||||
/bin/sh -c 'if test -e /etc/kubernetes/ssl/kube-ca.pem; then stat -c %U:%G /etc/kubernetes/ssl/kube-ca.pem; fi'
|
||||
```
|
||||
|
||||
**Expected result**:
|
||||
@@ -1700,7 +1700,7 @@ set the below parameter in `KUBELET_SYSTEM_PODS_ARGS` variable.
|
||||
``` bash
|
||||
--anonymous-auth=false
|
||||
```
|
||||
|
||||
|
||||
Based on your system, restart the kubelet service. For example:
|
||||
|
||||
``` bash
|
||||
@@ -1923,7 +1923,7 @@ systemctl restart kubelet.service
|
||||
'true' is equal to 'true'
|
||||
```
|
||||
|
||||
#### 4.2.7 Ensure that the `--make-iptables-util-chains` argument is set to `true` (Scored)
|
||||
#### 4.2.7 Ensure that the `--make-iptables-util-chains` argument is set to `true` (Scored)
|
||||
|
||||
**Result:** PASS
|
||||
|
||||
@@ -1935,7 +1935,7 @@ remove the `--make-iptables-util-chains` argument from the
|
||||
`KUBELET_SYSTEM_PODS_ARGS` variable.
|
||||
Based on your system, restart the kubelet service. For example:
|
||||
|
||||
```bash
|
||||
```bash
|
||||
systemctl daemon-reload
|
||||
systemctl restart kubelet.service
|
||||
```
|
||||
@@ -2088,7 +2088,7 @@ exit 0
|
||||
**Audit Execution:**
|
||||
|
||||
```
|
||||
./5.1.5.sh
|
||||
./5.1.5.sh
|
||||
```
|
||||
|
||||
**Expected result**:
|
||||
@@ -2215,7 +2215,7 @@ echo "pass"
|
||||
**Audit Execution:**
|
||||
|
||||
```
|
||||
./5.3.2.sh
|
||||
./5.3.2.sh
|
||||
```
|
||||
|
||||
**Expected result**:
|
||||
@@ -2255,7 +2255,7 @@ echo "--count=${default_resources}"
|
||||
**Audit Execution:**
|
||||
|
||||
```
|
||||
./5.6.4.sh
|
||||
./5.6.4.sh
|
||||
```
|
||||
|
||||
**Expected result**:
|
||||
@@ -2263,4 +2263,3 @@ echo "--count=${default_resources}"
|
||||
```
|
||||
'0' is equal to '0'
|
||||
```
|
||||
|
||||
+26
-26
@@ -1,26 +1,25 @@
|
||||
---
|
||||
title: Hardening Guide v2.4
|
||||
title: Hardening Guide with CIS 1.5 Benchmark
|
||||
weight: 99
|
||||
---
|
||||
|
||||
This document provides prescriptive guidance for hardening a production installation of Rancher v2.4. It outlines the configurations and controls required to address Kubernetes benchmark controls from the Center for Information Security (CIS).
|
||||
This document provides prescriptive guidance for hardening a production installation of a RKE cluster to be used with Rancher v2.5. It outlines the configurations and controls required to address Kubernetes benchmark controls from the Center for Information Security (CIS).
|
||||
|
||||
> This hardening guide describes how to secure the nodes in your cluster, and it is recommended to follow this guide before installing Kubernetes.
|
||||
|
||||
This hardening guide is intended to be used with specific versions of the CIS Kubernetes Benchmark, Kubernetes, and Rancher:
|
||||
This hardening guide is intended to be used for RKE clusters and associated with specific versions of the CIS Kubernetes Benchmark, Kubernetes, and Rancher:
|
||||
|
||||
Hardening Guide Version | Rancher Version | CIS Benchmark Version | Kubernetes Version
|
||||
------------------------|----------------|-----------------------|------------------
|
||||
Hardening Guide v2.4 | Rancher v2.4 | Benchmark v1.5 | Kubernetes 1.15
|
||||
Rancher Version | CIS Benchmark Version | Kubernetes Version
|
||||
----------------|-----------------------|------------------
|
||||
Rancher v2.5 | Benchmark v1.5 | Kubernetes 1.15
|
||||
|
||||
|
||||
[Click here to download a PDF version of this document](https://releases.rancher.com/documents/security/2.4/Rancher_Hardening_Guide.pdf)
|
||||
[Click here to download a PDF version of this document](https://releases.rancher.com/documents/security/2.5/Rancher_Hardening_Guide_CIS_1.5.pdf)
|
||||
|
||||
### Overview
|
||||
|
||||
This document provides prescriptive guidance for hardening a production installation of Rancher v2.4 with Kubernetes v1.15. It outlines the configurations required to address Kubernetes benchmark controls from the Center for Information Security (CIS).
|
||||
This document provides prescriptive guidance for hardening a RKE cluster to be used for installing Rancher v2.5 with Kubernetes v1.15 or provisioning a RKE cluster with Kubernetes 1.15 to be used within Rancher v2.5. It outlines the configurations required to address Kubernetes benchmark controls from the Center for Information Security (CIS).
|
||||
|
||||
For more detail about evaluating a hardened cluster against the official CIS benchmark, refer to the [CIS Benchmark Rancher Self-Assessment Guide - Rancher v2.4]({{< baseurl >}}/rancher/v2.x/en/security/benchmark-2.4/).
|
||||
For more detail about evaluating a hardened cluster against the official CIS benchmark, refer to the [CIS 1.5 Benchmark - Self-Assessment Guide - Rancher v2.5]({{< baseurl >}}/rancher/v2.x/en/security/rancher-2.5/1.5-benchmark-2.5/).
|
||||
|
||||
#### Known Issues
|
||||
|
||||
@@ -84,7 +83,7 @@ automountServiceAccountToken: false
|
||||
|
||||
Create a bash script file called `account_update.sh`. Be sure to `chmod +x account_update.sh` so the script has execute permissions.
|
||||
|
||||
```
|
||||
```
|
||||
#!/bin/bash -e
|
||||
|
||||
for namespace in $(kubectl get namespaces -A -o json | jq -r '.items[].metadata.name'); do
|
||||
@@ -147,6 +146,7 @@ done
|
||||
Execute this script to apply the `default-allow-all.yaml` the **permissive** `NetworkPolicy` to all namespaces.
|
||||
|
||||
### Reference Hardened RKE `cluster.yml` configuration
|
||||
|
||||
The reference `cluster.yml` is used by the RKE CLI that provides the configuration needed to achieve a hardened install
|
||||
of Rancher Kubernetes Engine (RKE). Install [documentation](https://rancher.com/docs/rke/latest/en/installation/) is
|
||||
provided with additional details about the configuration items. This reference `cluster.yml` does not include the required **nodes** directive which will vary depending on your environment. Documentation for node configuration can be found here: https://rancher.com/docs/rke/latest/en/config-options/nodes
|
||||
@@ -359,7 +359,7 @@ addons: |
|
||||
- kind: ServiceAccount
|
||||
name: tiller
|
||||
namespace: kube-system
|
||||
|
||||
|
||||
addons_include: []
|
||||
system_images:
|
||||
etcd: ""
|
||||
@@ -430,24 +430,24 @@ restore:
|
||||
dns: null
|
||||
```
|
||||
|
||||
### Reference Hardened RKE Template configuration
|
||||
### Reference Hardened RKE Template configuration
|
||||
|
||||
The reference RKE Template provides the configuration needed to achieve a hardened install of Kubenetes.
|
||||
RKE Templates are used to provision Kubernetes and define Rancher settings. Follow the Rancher
|
||||
[documentaion](https://rancher.com/docs/rancher/v2.x/en/installation) for additional installation and RKE Template details.
|
||||
|
||||
``` yaml
|
||||
#
|
||||
#
|
||||
# Cluster Config
|
||||
#
|
||||
#
|
||||
default_pod_security_policy_template_id: restricted
|
||||
docker_root_dir: /var/lib/docker
|
||||
enable_cluster_alerting: false
|
||||
enable_cluster_monitoring: false
|
||||
enable_network_policy: true
|
||||
#
|
||||
#
|
||||
# Rancher Config
|
||||
#
|
||||
#
|
||||
rancher_kubernetes_engine_config:
|
||||
addon_job_timeout: 30
|
||||
addons: |-
|
||||
@@ -602,32 +602,32 @@ rancher_kubernetes_engine_config:
|
||||
namespace: kube-system
|
||||
ignore_docker_version: true
|
||||
kubernetes_version: v1.15.9-rancher1-1
|
||||
#
|
||||
#
|
||||
# If you are using calico on AWS
|
||||
#
|
||||
#
|
||||
# network:
|
||||
# plugin: calico
|
||||
# calico_network_provider:
|
||||
# cloud_provider: aws
|
||||
#
|
||||
#
|
||||
# # To specify flannel interface
|
||||
#
|
||||
#
|
||||
# network:
|
||||
# plugin: flannel
|
||||
# flannel_network_provider:
|
||||
# iface: eth1
|
||||
#
|
||||
#
|
||||
# # To specify flannel interface for canal plugin
|
||||
#
|
||||
#
|
||||
# network:
|
||||
# plugin: canal
|
||||
# canal_network_provider:
|
||||
# iface: eth1
|
||||
#
|
||||
#
|
||||
network:
|
||||
mtu: 0
|
||||
plugin: canal
|
||||
#
|
||||
#
|
||||
# services:
|
||||
# kube-api:
|
||||
# service_cluster_ip_range: 10.43.0.0/16
|
||||
@@ -637,7 +637,7 @@ rancher_kubernetes_engine_config:
|
||||
# kubelet:
|
||||
# cluster_domain: cluster.local
|
||||
# cluster_dns_server: 10.43.0.10
|
||||
#
|
||||
#
|
||||
services:
|
||||
etcd:
|
||||
backup_config:
|
||||
@@ -0,0 +1,34 @@
|
||||
---
|
||||
title: Rancher v2.5
|
||||
weight: 1
|
||||
---
|
||||
|
||||
Rancher v2.5 introduced the capability to deploy Rancher on any Kubernetes cluster. For that reason, we now provide separate security hardening guides for Rancher deployments on each of Rancher's Kubernetes distributions.
|
||||
|
||||
Rancher has the following Kubernetes distributions:
|
||||
|
||||
- [**RKE,**]({{<baseurl>}}/rke/latest/en/) Rancher Kubernetes Engine, is a CNCF-certified Kubernetes distribution that runs entirely within Docker containers.
|
||||
- [**K3s,**]({{<baseurl>}}/k3s/latest/en/) is a fully conformant, lightweight Kubernetes distribution. It is easy to install, with half the memory of upstream Kubernetes, all in a binary of less than 100 MB.
|
||||
- [**RKE2**](https://docs.rke2.io/) is a fully conformant Kubernetes distribution that focuses on security and compliance within the U.S. Federal Government sector.
|
||||
|
||||
To harden a Kubernetes cluster outside of Rancher's distributions, refer to your Kubernetes provider docs.
|
||||
|
||||
# Guides
|
||||
|
||||
These guides have been tested along with the Rancher v2.5 release. Each self assessment guide is accompanied with a hardening guide and tested on a specific Kubernetes version and CIS benchmark version. If a CIS benchmark has not been validated for your Kubernetes version, you can choose to use the existing guides until a newer version is added.
|
||||
|
||||
### RKE Guides
|
||||
|
||||
Kubernetes Version | CIS Benchmark Version | Self Assessment Guide | Hardening Guides
|
||||
---|---|---|---
|
||||
Kubernetes v1.15+ | CIS v1.5 | [Link](./1.5-benchmark-2.5) | [Link](./1.5-hardening-2.5)
|
||||
|
||||
### RKE2 Guides
|
||||
|
||||
Kubernetes Version | CIS Benchmark Version | Self Assessment Guide | Hardening Guides
|
||||
---|---|---|---
|
||||
Kubernetes v1.18 | CIS v1.5 | [Link](https://docs.rke2.io/security/cis_self_assessment/) | [Link](https://docs.rke2.io/security/hardening_guide/)
|
||||
|
||||
### K3s Guides
|
||||
|
||||
The K3s security guides will be added soon.
|
||||
@@ -1,5 +1,5 @@
|
||||
---
|
||||
title: Imported clusters
|
||||
title: Registered clusters
|
||||
weight: 105
|
||||
---
|
||||
|
||||
@@ -15,6 +15,8 @@ If the cattle-cluster-agent cannot connect to the configured `server-url`, the c
|
||||
|
||||
#### cattle-node-agent
|
||||
|
||||
> Note: Starting in Rancher 2.5 cattle-node-agents are only present in clusters created in Rancher with RKE.
|
||||
|
||||
Check if the cattle-node-agent pods are present on each node, have status **Running** and don't have a high count of Restarts:
|
||||
|
||||
```
|
||||
|
||||
@@ -10,14 +10,13 @@ Make sure you configured the correct kubeconfig (for example, `export KUBECONFIG
|
||||
### Double check if all the required ports are opened in your (host) firewall
|
||||
|
||||
Double check if all the [required ports]({{<baseurl>}}/rancher/v2.x/en/cluster-provisioning/node-requirements/#networking-requirements/) are opened in your (host) firewall. The overlay network uses UDP in comparison to all other required ports which are TCP.
|
||||
|
||||
### Check if overlay network is functioning correctly
|
||||
|
||||
The pod can be scheduled to any of the hosts you used for your cluster, but that means that the NGINX ingress controller needs to be able to route the request from `NODE_1` to `NODE_2`. This happens over the overlay network. If the overlay network is not functioning, you will experience intermittent TCP/HTTP connection failures due to the NGINX ingress controller not being able to route to the pod.
|
||||
|
||||
To test the overlay network, you can launch the following `DaemonSet` definition. This will run a `busybox` container on every host, which we will use to run a `ping` test between containers on all hosts.
|
||||
To test the overlay network, you can launch the following `DaemonSet` definition. This will run a `swiss-army-knife` container on every host (image was developed by Rancher engineers and can be found here: https://github.com/leodotcloud/swiss-army-knife), which we will use to run a `ping` test between containers on all hosts.
|
||||
|
||||
1. Save the following file as `ds-overlaytest.yml`
|
||||
1. Save the following file as `overlaytest.yml`
|
||||
|
||||
```
|
||||
apiVersion: apps/v1
|
||||
@@ -36,46 +35,58 @@ To test the overlay network, you can launch the following `DaemonSet` definition
|
||||
tolerations:
|
||||
- operator: Exists
|
||||
containers:
|
||||
- image: busybox:1.28
|
||||
- image: leodotcloud/swiss-army-knife
|
||||
imagePullPolicy: Always
|
||||
name: busybox
|
||||
name: overlaytest
|
||||
command: ["sh", "-c", "tail -f /dev/null"]
|
||||
terminationMessagePath: /dev/termination-log
|
||||
|
||||
```
|
||||
|
||||
2. Launch it using `kubectl create -f ds-overlaytest.yml`
|
||||
2. Launch it using `kubectl create -f overlaytest.yml`
|
||||
3. Wait until `kubectl rollout status ds/overlaytest -w` returns: `daemon set "overlaytest" successfully rolled out`.
|
||||
4. Run the following command, from the same location, to let each container on every host ping each other (it's a single line bash command).
|
||||
|
||||
4. Run the following script, from the same location. It will have each `overlaytest` container on every host ping each other:
|
||||
```
|
||||
echo "=> Start network overlay test"; kubectl get pods -l name=overlaytest -o jsonpath='{range .items[*]}{@.metadata.name}{" "}{@.spec.nodeName}{"\n"}{end}' | while read spod shost; do kubectl get pods -l name=overlaytest -o jsonpath='{range .items[*]}{@.status.podIP}{" "}{@.spec.nodeName}{"\n"}{end}' | while read tip thost; do kubectl --request-timeout='10s' exec $spod -- /bin/sh -c "ping -c2 $tip > /dev/null 2>&1"; RC=$?; if [ $RC -ne 0 ]; then echo $shost cannot reach $thost; fi; done; done; echo "=> End network overlay test"
|
||||
#!/bin/bash
|
||||
echo "=> Start network overlay test"
|
||||
kubectl get pods -l name=overlaytest -o jsonpath='{range .items[*]}{@.metadata.name}{" "}{@.spec.nodeName}{"\n"}{end}' |
|
||||
while read spod shost
|
||||
do kubectl get pods -l name=overlaytest -o jsonpath='{range .items[*]}{@.status.podIP}{" "}{@.spec.nodeName}{"\n"}{end}' |
|
||||
while read tip thost
|
||||
do kubectl --request-timeout='10s' exec $spod -c overlaytest -- /bin/sh -c "ping -c2 $tip > /dev/null 2>&1"
|
||||
RC=$?
|
||||
if [ $RC -ne 0 ]
|
||||
then echo FAIL: $spod on $shost cannot reach pod IP $tip on $thost
|
||||
else echo $shost can reach $thost
|
||||
fi
|
||||
done
|
||||
done
|
||||
echo "=> End network overlay test"
|
||||
```
|
||||
|
||||
5. When this command has finished running, the output indicating everything is correct is:
|
||||
5. When this command has finished running, it will output the state of each route:
|
||||
|
||||
```
|
||||
=> Start network overlay test
|
||||
Error from server (NotFound): pods "wk2" not found
|
||||
FAIL: overlaytest-5bglp on wk2 cannot reach pod IP 10.42.7.3 on wk2
|
||||
Error from server (NotFound): pods "wk2" not found
|
||||
FAIL: overlaytest-5bglp on wk2 cannot reach pod IP 10.42.0.5 on cp1
|
||||
Error from server (NotFound): pods "wk2" not found
|
||||
FAIL: overlaytest-5bglp on wk2 cannot reach pod IP 10.42.2.12 on wk1
|
||||
command terminated with exit code 1
|
||||
FAIL: overlaytest-v4qkl on cp1 cannot reach pod IP 10.42.7.3 on wk2
|
||||
cp1 can reach cp1
|
||||
cp1 can reach wk1
|
||||
command terminated with exit code 1
|
||||
FAIL: overlaytest-xpxwp on wk1 cannot reach pod IP 10.42.7.3 on wk2
|
||||
wk1 can reach cp1
|
||||
wk1 can reach wk1
|
||||
=> End network overlay test
|
||||
```
|
||||
If you see error in the output, there is some issue with the route between the pods on the two hosts. In the above output the node `wk2` has no connectivity over the overlay network. This could be because the [required ports]({{<baseurl>}}/rancher/v2.x/en/cluster-provisioning/node-requirements/#networking-requirements/) for overlay networking are not opened for `wk2`.
|
||||
6. You can now clean up the DaemonSet by running `kubectl delete ds/overlaytest`.
|
||||
|
||||
If you see error in the output, that means that the [required ports]({{<baseurl>}}/rancher/v2.x/en/cluster-provisioning/node-requirements/#networking-requirements/) for overlay networking are not opened between the hosts indicated.
|
||||
|
||||
Example error output of a situation where NODE1 had the UDP ports blocked.
|
||||
|
||||
```
|
||||
=> Start network overlay test
|
||||
command terminated with exit code 1
|
||||
NODE2 cannot reach NODE1
|
||||
command terminated with exit code 1
|
||||
NODE3 cannot reach NODE1
|
||||
command terminated with exit code 1
|
||||
NODE1 cannot reach NODE2
|
||||
command terminated with exit code 1
|
||||
NODE1 cannot reach NODE3
|
||||
=> End network overlay test
|
||||
```
|
||||
|
||||
Cleanup the busybox DaemonSet by running `kubectl delete ds/overlaytest`.
|
||||
|
||||
### Check if MTU is correctly configured on hosts and on peering/tunnel appliances/devices
|
||||
|
||||
|
||||
@@ -35,6 +35,10 @@ The snapshot is stored in `/opt/rke/etcd-snapshots`. If the directory is configu
|
||||
|
||||
In the case when multiple etcd nodes exist, any created snapshot is created after the cluster has been health checked, so it can be considered a valid snapshot of the data in the etcd cluster.
|
||||
|
||||
_Available as of v1.1.4_
|
||||
|
||||
Each snapshot will include the cluster state file in addition to the etcd snapshot file.
|
||||
|
||||
### Snapshot Naming
|
||||
|
||||
The name of the snapshot is auto-generated. The `--name` option can be used to override the name of the snapshot when creating one-time snapshots with the RKE CLI.
|
||||
@@ -46,10 +50,11 @@ An example one-time snapshot name is `rke_etcd_snapshot_2020-10-15T16:47:24+02:0
|
||||
On restore, the following process is used:
|
||||
|
||||
1. The snapshot is retrieved from S3, if S3 is configured.
|
||||
2. The snapshot is unzipped (if zipped).
|
||||
3. One of the etcd nodes in the cluster serves that snapshot file to the other nodes.
|
||||
4. The other etcd nodes download the snapshot and validate the checksum so that they all use the same snapshot for the restore.
|
||||
5. The cluster is restored and post-restore actions will be done in the cluster.
|
||||
1. The snapshot is unzipped (if zipped).
|
||||
1. It is checked if the cluster state file is included in the snapshot, if it is included, it will be used instead of the local cluster state file (_Available as of v1.1.4_)
|
||||
1. One of the etcd nodes in the cluster serves that snapshot file to the other nodes.
|
||||
1. The other etcd nodes download the snapshot and validate the checksum so that they all use the same snapshot for the restore.
|
||||
1. The cluster is restored and post-restore actions will be done in the cluster.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
|
||||
@@ -73,7 +73,7 @@ nodes:
|
||||
|
||||
### 4. Restore etcd on the New Node from the Backup
|
||||
|
||||
> **Prerequisite:** Ensure your `cluster.rkestate` is present before starting the restore, because this contains your certificate data for the cluster.
|
||||
> **Prerequisite:** If the snapshot was created using RKE v1.1.4 or higher, the cluster state file should be included in the snapshot. The cluster state file will be automatically extracted and used for the restore. If the snapshot was created using RKE v1.1.3 or lower, please ensure your `cluster.rkestate` is present before starting the restore, because this contains your certificate data for the cluster.
|
||||
|
||||
After the new node is added to the `cluster.yml`, run the `rke etcd snapshot-restore` to launch `etcd` from the backup:
|
||||
|
||||
|
||||
@@ -23,6 +23,10 @@ The following actions will be performed when you run the command:
|
||||
|
||||
The snapshot used to restore your etcd cluster can either be stored locally in `/opt/rke/etcd-snapshots` or from a S3 compatible backend.
|
||||
|
||||
_Available as of v1.1.4_
|
||||
|
||||
If the snapshot contains the cluster state file, it will automatically be extracted and used for the restore. If you want to force the use of the local state file, you can add `--use-local-state` to the command. If the snapshot was created using an RKE version before v1.1.4, or if the snapshot does not contain a state file, make sure the cluster state file (by default available as `cluster.rkestate`) is present before executing the command.
|
||||
|
||||
### Example of Restoring from a Local Snapshot
|
||||
|
||||
To restore etcd from a local snapshot, run:
|
||||
@@ -37,8 +41,6 @@ The snapshot is assumed to be located in `/opt/rke/etcd-snapshots`.
|
||||
|
||||
### Example of Restoring from a Snapshot in S3
|
||||
|
||||
> **Prerequisite:** Ensure your `cluster.rkestate` is present before starting the restore, because this contains your certificate data for the cluster.
|
||||
|
||||
When restoring etcd from a snapshot located in S3, the command needs the S3 information in order to connect to the S3 backend and retrieve the snapshot.
|
||||
|
||||
```shell
|
||||
@@ -60,6 +62,7 @@ $ rke etcd snapshot-restore \
|
||||
| --- | --- | ---|
|
||||
| `--name` value | Specify snapshot name | |
|
||||
| `--config` value | Specify an alternate cluster YAML file (default: `cluster.yml`) [$RKE_CONFIG] | |
|
||||
| `--use-local-state` | Force the use of the local state file instead of looking for a state file in the snapshot _Available as of v1.1.4_ | |
|
||||
| `--s3` | Enabled backup to s3 |* |
|
||||
| `--s3-endpoint` value | Specify s3 endpoint url (default: "s3.amazonaws.com") | * |
|
||||
| `--access-key` value | Specify s3 accessKey | *|
|
||||
|
||||
@@ -8,7 +8,7 @@ weight: 5
|
||||
|
||||
- [Operating System](#operating-system)
|
||||
- [General Linux Requirements](#general-linux-requirements)
|
||||
- [Red Hat Enterprise Linux (RHEL) / Oracle Enterprise Linux (OEL) / CentOS](#red-hat-enterprise-linux-rhel-oracle-enterprise-linux-oel-centos)
|
||||
- [Red Hat Enterprise Linux (RHEL) / Oracle Linux (OL) / CentOS](#red-hat-enterprise-linux-rhel-oracle-enterprise-linux-ol-centos)
|
||||
|
||||
- [Using upstream Docker](#using-upstream-docker)
|
||||
- [Using RHEL/CentOS packaged Docker](#using-rhel-centos-packaged-docker)
|
||||
@@ -98,9 +98,9 @@ xt_tcpudp |
|
||||
net.bridge.bridge-nf-call-iptables=1
|
||||
```
|
||||
|
||||
### Red Hat Enterprise Linux (RHEL) / Oracle Enterprise Linux (OEL) / CentOS
|
||||
### Red Hat Enterprise Linux (RHEL) / Oracle Linux (OL) / CentOS
|
||||
|
||||
If using Red Hat Enterprise Linux, Oracle Enterprise Linux or CentOS, you cannot use the `root` user as [SSH user]({{<baseurl>}}/rke/latest/en/config-options/nodes/#ssh-user) due to [Bugzilla 1527565](https://bugzilla.redhat.com/show_bug.cgi?id=1527565). Please follow the instructions below how to setup Docker correctly, based on the way you installed Docker on the node.
|
||||
If using Red Hat Enterprise Linux, Oracle Linux or CentOS, you cannot use the `root` user as [SSH user]({{<baseurl>}}/rke/latest/en/config-options/nodes/#ssh-user) due to [Bugzilla 1527565](https://bugzilla.redhat.com/show_bug.cgi?id=1527565). Please follow the instructions below how to setup Docker correctly, based on the way you installed Docker on the node.
|
||||
|
||||
#### Using upstream Docker
|
||||
If you are using upstream Docker, the package name is `docker-ce` or `docker-ee`. You can check the installed package by executing:
|
||||
|
||||
@@ -1,17 +0,0 @@
|
||||
<div>
|
||||
<p>Rancher uses <code>etcd</code> as datastore. When using the Docker Install, the embedded <code>etcd</code> is
|
||||
being used. The persistent data is at the following path in the container: <code>/var/lib/rancher</code>. You can
|
||||
bind mount a host volume to this location to preserve data on the host it is running on. When using RancherOS,
|
||||
please check what <a
|
||||
href="/docs/os/v1.x/en/installation/system-services/system-docker-volumes/#user-volumes">persistent storage
|
||||
directories</a> you can use to store the data.</p>
|
||||
|
||||
<p>Command:</p>
|
||||
|
||||
<pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4">
|
||||
docker run -d --restart=unless-stopped \
|
||||
-p 80:80 -p 443:443 \
|
||||
-v /opt/rancher:/var/lib/rancher \
|
||||
rancher/rancher:latest
|
||||
</pre>
|
||||
</div>
|
||||
Reference in New Issue
Block a user