public/rancher-docs
1
0
Fork 0
mirror of https://github.com/rancher/rancher-docs.git synced 2026-09-11 03:35:49 +00:00
Code Issues Packages Projects Releases Wiki Activity

Update the hardening guides for rancher 2.7 (#453)

Browse Source
This commit is contained in:
Jiaqi Luo
2023-03-03 11:28:50 -07:00
committed by GitHub
parent 21f1cf2373
commit cf9222b1ad
28 changed files with 29664 additions and 55526 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.gitignore
+3 -1
View File
@@ -17,4 +17,6 @@
npm-debug.log*
yarn-debug.log*
yarn-error.log*
yarn-error.log*
.idea/
docs/pages-for-subheaders/cis-scans.md
+32 -22
View File
@@ -13,8 +13,7 @@ The Center for Internet Security is a 501(c\)(3) non-profit organization, formed
CIS Benchmarks are best practices for the secure configuration of a target system. CIS Benchmarks are developed through the generous volunteer efforts of subject matter experts, technology vendors, public and private community members, and the CIS Benchmark Development team.
The official Benchmark documents are available through the CIS website. The sign-up form to access the documents is
<a href="https://learn.cisecurity.org/benchmarks" target="_blank">here.</a>
[Sign up](https://learn.cisecurity.org/benchmarks) at the CIS website to view the official Benchmark documents.
## About the Generated Report
@@ -30,20 +29,20 @@ Some tests are designated as "Not Applicable." These tests will not be run on an
The report contains the following information:
| Column in Report | Description |
|------------------|-------------|
| `id` | The ID number of the CIS Benchmark. |
| `description` | The description of the CIS Benchmark test. |
| `remediation` | What needs to be fixed in order to pass the test. |
| `state` | Indicates if the test passed, failed, was skipped, or was not applicable. |
| `node_type` | The node role, which affects which tests are run on the node. Master tests are run on controlplane nodes, etcd tests are run on etcd nodes, and node tests are run on the worker nodes. |
| `audit` | This is the audit check that `kube-bench` runs for this test. |
| `audit_config` | Any configuration applicable to the audit script. |
| `test_info` | Test-related info as reported by `kube-bench`, if any. |
| `commands` | Test-related commands as reported by `kube-bench`, if any. |
| `config_commands` | Test-related configuration data as reported by `kube-bench`, if any. |
| `actual_value` | The test's actual value, present if reported by `kube-bench`. |
| `expected_result` | The test's expected result, present if reported by `kube-bench`. |
| Column in Report | Description |
|-------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `id` | The ID number of the CIS Benchmark. |
| `description` | The description of the CIS Benchmark test. |
| `remediation` | What needs to be fixed in order to pass the test. |
| `state` | Indicates if the test passed, failed, was skipped, or was not applicable. |
| `node_type` | The node role, which affects which tests are run on the node. Master tests are run on controlplane nodes, etcd tests are run on etcd nodes, and node tests are run on the worker nodes. |
| `audit` | This is the audit check that `kube-bench` runs for this test. |
| `audit_config` | Any configuration applicable to the audit script. |
| `test_info` | Test-related info as reported by `kube-bench`, if any. |
| `commands` | Test-related commands as reported by `kube-bench`, if any. |
| `config_commands` | Test-related configuration data as reported by `kube-bench`, if any. |
| `actual_value` | The test's actual value, present if reported by `kube-bench`. |
| `expected_result` | The test's expected result, present if reported by `kube-bench`. |
Refer to [the table in the cluster hardening guide](./rancher-security.md) for information on which versions of Kubernetes, the Benchmark, Rancher, and our cluster hardening guide correspond to each other. Also refer to the hardening guide for configuration files of CIS-compliant clusters and information on remediating failed tests.
@@ -51,16 +50,27 @@ Refer to [the table in the cluster hardening guide](./rancher-security.md) for i
The following profiles are available:
- Generic CIS 1.5
- Generic CIS 1.6
- RKE permissive 1.5
- RKE hardened 1.5
- Generic CIS 1.20
- Generic CIS 1.23
- RKE permissive 1.6
- RKE hardened 1.6
- RKE2 permissive 1.5
- RKE2 hardened 1.5
- RKE permissive 1.20
- RKE hardened 1.20
- RKE permissive 1.23
- RKE hardened 1.23
- RKE2 permissive 1.6
- RKE2 hardened 1.6
- RKE2 permissive 1.20
- RKE2 hardened 1.20
- RKE2 permissive 1.23
- RKE2 hardened 1.23
- K3s permissive 1.6
- K3s hardened 1.6
- K3s permissive 1.20
- K3s hardened 1.20
- K3s permissive 1.23
- K3s hardened 1.23
- AKS
- EKS
- GKE
@@ -105,4 +115,4 @@ For more information about configuring the custom resources for the scans, profi
## How-to Guides
Please refer [here](../pages-for-subheaders/cis-scan-guides.md) for how-to guides on CIS scans.
Please refer to the [CIS Scan Guides](../pages-for-subheaders/cis-scan-guides.md) to learn how to run CIS scans.
docs/pages-for-subheaders/rancher-hardening-guides.md
+52
View File
@@ -0,0 +1,52 @@
---
title: Self-Assessment and Hardening Guides for Rancher
---
Rancher provides specific security hardening guides for each supported Rancher version's Kubernetes distributions.
## Rancher Kubernetes Distributions
Rancher uses the following Kubernetes distributions:
- [**RKE**](https://rancher.com/docs/rke/latest/en/), Rancher Kubernetes Engine, is a CNCF-certified Kubernetes distribution that runs entirely within Docker containers.
- [**RKE2**](https://docs.rke2.io/) is a fully conformant Kubernetes distribution that focuses on security and compliance within the U.S. Federal Government sector.
- [**K3s**](https://docs.k3s.io/) is a fully conformant, lightweight Kubernetes distribution. It is easy to install, with half the memory requirement of upstream Kubernetes, all in a binary of less than 100 MB.
To harden a Kubernetes cluster that's running a distribution other than those listed, refer to your Kubernetes provider docs.
## Hardening Guides and Benchmark Versions
Each self-assessment guide is accompanied by a hardening guide. These guides were tested alongside the listed Rancher releases. Each self-assessment guides was tested on a specific Kubernetes version and CIS benchmark version. If a CIS benchmark has not been validated for your Kubernetes version, you can use the existing guides until a guide for your version is added.
### RKE Guides
| Kubernetes Version | CIS Benchmark Version | Self Assessment Guide | Hardening Guides |
|------------------------------|-----------------------|----------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------|
| Kubernetes v1.18 up to v1.23 | CIS v1.6 | [Link](../reference-guides/rancher-security/hardening-guides/rke1-self-assessment-guide-with-cis-v1.6-benchmark.md) | [Link](../reference-guides/rancher-security/hardening-guides/rke1-hardening-guide.md) |
| Kubernetes v1.19 up to v1.21 | CIS v1.20 | [Link](../reference-guides/rancher-security/hardening-guides/rke1-self-assessment-guide-with-cis-v1.20-benchmark.md) | [Link](../reference-guides/rancher-security/hardening-guides/rke1-hardening-guide.md) |
| Kubernetes v1.22 up to v1.25 | CIS v1.23 | [Link](../reference-guides/rancher-security/hardening-guides/rke1-self-assessment-guide-with-cis-v1.23-k8s-v1.24.md) | [Link](../reference-guides/rancher-security/hardening-guides/rke1-hardening-guide.md) |
### RKE2 Guides
| Type | Kubernetes Version | CIS Benchmark Version | Self Assessment Guide | Hardening Guides |
|----------------------------------|------------------------------|-----------------------|----------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------|
| Rancher provisioned RKE2 cluster | Kubernetes v1.21 up to v1.23 | CIS v1.6 | [Link](../reference-guides/rancher-security/hardening-guides/rke2-self-assessment-guide-with-cis-v1.6-benchmark.md) | [Link](../reference-guides/rancher-security/hardening-guides/rke2-hardening-guide.md) |
| Rancher provisioned RKE2 cluster | Kubernetes v1.22 up to v1.25 | CIS v1.23 | [Link](../reference-guides/rancher-security/hardening-guides/rke2-self-assessment-guide-with-cis-v1.23-benchmark.md) | [Link](../reference-guides/rancher-security/hardening-guides/rke2-hardening-guide-with-cis-v1.23-benchmark.md) |
| Standalone RKE2 | Kubernetes v1.20 | CIS v1.6 | [Link](https://docs.rke2.io/security/cis_self_assessment16/) | [Link](https://docs.rke2.io/security/hardening_guide/) |
| Standalone RKE2 | Kubernetes v1.25 | CIS v1.23 | [Link](https://docs.rke2.io/security/cis_self_assessment123/) | [Link](https://docs.rke2.io/security/hardening_guide/) |
### K3s Guides
| Type | Kubernetes Version | CIS Benchmark Version | Self Assessment Guide | Hardening Guides |
|---------------------------------|------------------------------|-----------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------|
| Rancher provisioned K3s cluster | Kubernetes v1.21 | CIS v1.20 | [CIS v1.20](../reference-guides/rancher-security/hardening-guides/k3s-self-assessment-guide-with-cis-v1.20-benchmark.md) and [CIS v1.23](../reference-guides/rancher-security/hardening-guides/k3s-self-assessment-guide-with-cis-v1.23-k8s-v1.23.md) | [Link](../reference-guides/rancher-security/hardening-guides/k3s-hardening-guide.md) |
| Rancher provisioned K3s cluster | Kubernetes v1.22 up to v1.24 | CIS v1.23 | [CIS v1.20](../reference-guides/rancher-security/hardening-guides/k3s-self-assessment-guide-with-cis-v1.20-benchmark.md) and [CIS v1.23](../reference-guides/rancher-security/hardening-guides/k3s-self-assessment-guide-with-cis-v1.23-k8s-v1.23.md) | [Link](../reference-guides/rancher-security/hardening-guides/k3s-hardening-guide.md) |
| Standalone K3s | Kubernetes v1.22 up to v1.24 | CIS v1.23 | [Link](https://docs.k3s.io/security/self-assessment) | [Link](https://docs.k3s.io/security/hardening-guide) |
## Rancher with SELinux
[Security-Enhanced Linux (SELinux)](https://en.wikipedia.org/wiki/Security-Enhanced_Linux) is a kernel module that adds extra access controls and security tools to Linux. Historically used by government agencies, SELinux is now industry-standard. SELinux is enabled by default on RHEL and CentOS.
To use Rancher with SELinux, we recommend [installing](selinux-rpm.md#installing-the-rancher-selinux-rpm) the `rancher-selinux` RPM.
docs/pages-for-subheaders/rancher-v2.6-hardening-guides.md
-51
View File
@@ -1,51 +0,0 @@
---
title: Self-Assessment and Hardening Guides for Rancher v2.6
---
Rancher provides specific security hardening guides for each supported Rancher's Kubernetes distributions.
## Rancher Kubernetes Distributions
Rancher uses the following Kubernetes distributions:
- [**RKE**](https://rancher.com/docs/rke/latest/en/), Rancher Kubernetes Engine, is a CNCF-certified Kubernetes distribution that runs entirely within Docker containers.
- [**RKE2**](https://docs.rke2.io/) is a fully conformant Kubernetes distribution that focuses on security and compliance within the U.S. Federal Government sector.
- [**K3s**](https://rancher.com/docs/k3s/latest/en/) is a fully conformant, lightweight Kubernetes distribution. It is easy to install, with half the memory of upstream Kubernetes, all in a binary of less than 100 MB.
To harden a Kubernetes cluster outside of Rancher's distributions, refer to your Kubernetes provider docs.
## Hardening Guides and Benchmark Versions
These guides have been tested along with the Rancher v2.6 release. Each self-assessment guide is accompanied with a hardening guide and tested on a specific Kubernetes version and CIS benchmark version. If a CIS benchmark has not been validated for your Kubernetes version, you can choose to use the existing guides until a newer version is added.
### RKE Guides
| Kubernetes Version | CIS Benchmark Version | Self Assessment Guide | Hardening Guides |
| ------------------ | --------------------- | --------------------- | ---------------- |
| Kubernetes v1.18 up to v1.23 | CIS v1.6 | [Link](../reference-guides/rancher-security/rancher-v2.6-hardening-guides/rke1-self-assessment-guide-with-cis-v1.6-benchmark.md) | [Link](../reference-guides/rancher-security/rancher-v2.6-hardening-guides/rke1-hardening-guide-with-cis-v1.6-benchmark.md) |
:::note
- CIS v1.20 benchmark version for Kubernetes v1.19 and v1.20 is not yet released as a profile in Rancher's CIS Benchmark chart.
:::
### RKE2 Guides
| Type | Kubernetes Version | CIS Benchmark Version | Self Assessment Guide | Hardening Guides |
| ---- | ------------------ | --------------------- | --------------------- | ---------------- |
| Rancher provisioned RKE2 cluster | Kubernetes v1.21 up to v1.23 | CIS v1.6 | [Link](../reference-guides/rancher-security/rancher-v2.6-hardening-guides/rke2-self-assessment-guide-with-cis-v1.6-benchmark.md) | [Link](../reference-guides/rancher-security/rancher-v2.6-hardening-guides/rke2-hardening-guide-with-cis-v1.6-benchmark.md) |
| Standalone RKE2 | Kubernetes v1.21 up to v1.23 | CIS v1.6 | [Link](https://docs.rke2.io/security/cis_self_assessment16/) | [Link](https://docs.rke2.io/security/hardening_guide/) |
### K3s Guides
| Type | Kubernetes Version | CIS Benchmark Version | Self Assessment Guide | Hardening Guides |
| ---- | ------------------ | --------------------- | --------------------- | ---------------- |
| Standalone K3s | Kubernetes v1.21 and v1.22 | CIS v1.6 | [Link](https://docs.k3s.io/security/self-assessment) | [Link](https://docs.k3s.io/security/hardening-guide) |
## Rancher with SELinux
[Security-Enhanced Linux (SELinux)](https://en.wikipedia.org/wiki/Security-Enhanced_Linux) is a security enhancement to Linux. After being historically used by government agencies, SELinux is now industry standard and is enabled by default on RHEL and CentOS.
To use Rancher with SELinux, we recommend installing the `rancher-selinux` RPM according to the instructions on [this page.](selinux-rpm.md#installing-the-rancher-selinux-rpm)
docs/pages-for-subheaders/rancher-v2.7-hardening-guides.md
-52
View File
@@ -1,52 +0,0 @@
---
title: Self-Assessment and Hardening Guides for Rancher v2.7
---
Rancher provides specific security hardening guides for each supported Rancher's Kubernetes distributions.
## Rancher Kubernetes Distributions
Rancher uses the following Kubernetes distributions:
- [**RKE**](https://rancher.com/docs/rke/latest/en/), Rancher Kubernetes Engine, is a CNCF-certified Kubernetes distribution that runs entirely within Docker containers.
- [**RKE2**](https://docs.rke2.io/) is a fully conformant Kubernetes distribution that focuses on security and compliance within the U.S. Federal Government sector.
- [**K3s**](https://rancher.com/docs/k3s/latest/en/) is a fully conformant, lightweight Kubernetes distribution. It is easy to install, with half the memory of upstream Kubernetes, all in a binary of less than 100 MB.
To harden a Kubernetes cluster outside of Rancher's distributions, refer to your Kubernetes provider docs.
## Hardening Guides and Benchmark Versions
These guides have been tested along with the Rancher v2.7 release. Each self-assessment guide is accompanied with a hardening guide and tested on a specific Kubernetes version and CIS benchmark version. If a CIS benchmark has not been validated for your Kubernetes version, you can choose to use the existing guides until a newer version is added.
### RKE Guides
| Kubernetes Version | CIS Benchmark Version | Self Assessment Guide | Hardening Guides |
| ------------------ | --------------------- | --------------------- | ---------------- |
| Kubernetes v1.18 up to v1.23 | CIS v1.6 | [Link](../reference-guides/rancher-security/rancher-v2.6-hardening-guides/rke1-self-assessment-guide-with-cis-v1.6-benchmark.md) | [Link](../reference-guides/rancher-security/rancher-v2.6-hardening-guides/rke1-hardening-guide-with-cis-v1.6-benchmark.md) |
:::note
- CIS v1.20 benchmark version for Kubernetes v1.19 and v1.20 is not yet released as a profile in Rancher's CIS Benchmark chart.
:::
### RKE2 Guides
| Type | Kubernetes Version | CIS Benchmark Version | Self Assessment Guide | Hardening Guides |
| ---- | ------------------ | --------------------- | --------------------- | ---------------- |
| Rancher provisioned RKE2 cluster | Kubernetes v1.21 up to v1.23 | CIS v1.6 | [Link](../reference-guides/rancher-security/rancher-v2.6-hardening-guides/rke2-self-assessment-guide-with-cis-v1.6-benchmark.md) | [Link](../reference-guides/rancher-security/rancher-v2.6-hardening-guides/rke2-hardening-guide-with-cis-v1.6-benchmark.md) |
| Standalone RKE2 | Kubernetes v1.21 up to v1.23 | CIS v1.6 | [Link](https://docs.rke2.io/security/cis_self_assessment16/) | [Link](https://docs.rke2.io/security/hardening_guide/) |
### K3s Guides
| Type | Kubernetes Version | CIS Benchmark Version | Self Assessment Guide | Hardening Guides |
| ---- | ------------------ | --------------------- | --------------------- | ---------------- |
| Rancher provisioned K3s cluster | Kubernetes v1.21 up to v1.24 | CIS v1.20 and v1.23 | [CIS v1.20](../reference-guides/rancher-security/rancher-v2.7-hardening-guides/k3s-self-assessment-guide-with-cis-v1.20-benchmark.md) and [CIS v1.23](../reference-guides/rancher-security/rancher-v2.7-hardening-guides/k3s-self-assessment-guide-with-cis-v1.23-benchmark.md) | [Link](../reference-guides/rancher-security/rancher-v2.7-hardening-guides/k3s-hardening-guide-with-cis-benchmark.md) |
| Standalone K3s | Kubernetes v1.21 and v1.22 | CIS v1.6 | [Link](https://docs.k3s.io/security/self-assessment) | [Link](https://docs.k3s.io/security/hardening-guide) |
## Rancher with SELinux
[Security-Enhanced Linux (SELinux)](https://en.wikipedia.org/wiki/Security-Enhanced_Linux) is a security enhancement to Linux. After being historically used by government agencies, SELinux is now industry standard and is enabled by default on RHEL and CentOS.
To use Rancher with SELinux, we recommend installing the `rancher-selinux` RPM according to the instructions on [this page.](selinux-rpm.md#installing-the-rancher-selinux-rpm)
docs/reference-guides/rancher-security/hardening-guides/k3s-hardening-guide.md
+638
View File
@@ -0,0 +1,638 @@
---
title: K3s Hardening Guide
---
This document provides prescriptive guidance for how to harden a K3s cluster intended for production, before provisioning it with Rancher. It outlines the configurations and controls required for Center for Information Security (CIS) Kubernetes benchmark controls.
:::note
This hardening guide describes how to secure the nodes in your cluster. We recommended that you follow this guide before you install Kubernetes.
:::
This hardening guide is intended to be used for K3s clusters and associated with the following versions of the CIS Kubernetes Benchmark, Kubernetes, and Rancher:
| Rancher Version | CIS Benchmark Version | Kubernetes Version |
|-----------------|-----------------------|------------------------------|
| Rancher v2.7 | Benchmark v1.23 | Kubernetes v1.22 up to v1.25 |
For more details on how to evaluate a hardened K3s cluster against the official CIS benchmark, refer to the K3s self-assessment guides for specific CIS benchmark versions.
K3s passes a number of the Kubernetes CIS controls without modification, as it applies several security mitigations by default. There are some notable exceptions to this that require manual intervention to fully comply with the CIS Benchmark:
1. K3s doesn't modify the host operating system. Any host-level modifications need to be done manually.
2. Certain CIS policy controls for `NetworkPolicies` and `PodSecurityStandards` (`PodSecurityPolicies` on v1.24 and older) restrict cluster functionality.
You must opt into having K3s configure these policies. Add the appropriate options to your command-line flags or configuration file (enable admission plugins), and manually apply the appropriate policies.
See further for more details.
The first section (1.1) of the CIS Benchmark primarily focuses on pod manifest permissions and ownership. Since everything in the distribution is packaged in a single binary, this section doesn't apply to the core components of K3s.
## Host-level Requirements
### Set kernel parameters
The following `sysctl` configuration is recommended for all nodes type in the cluster. Set the following parameters in `/etc/sysctl.d/90-kubelet.conf`:
```ini
vm.panic_on_oom=0
vm.overcommit_memory=1
kernel.panic=10
kernel.panic_on_oops=1
```
Run `sudo sysctl -p /etc/sysctl.d/90-kubelet.conf` to enable the settings.
## Kubernetes Runtime Requirements
The CIS Benchmark runtime requirements center around pod security (via PSP or PSA), network policies and API Server auditing logs.
### Ensure `protect-kernel-defaults` is set
This is a kubelet flag that will cause the kubelet to exit if the required kernel parameters are unset or are set to values that are different from the kubelet's defaults.
The `protect-kernel-defaults` flag can be set in the cluster configuration in Rancher.
```yaml
spec:
rkeConfig:
machineSelectorConfig:
- config:
protect-kernel-defaults: true
```
### Pod Security
By default, K3s doesn't include any pod security or network policies. However, K3s ships with a controller that enforces any network policies you create. K3s doesn't enable auditing by default, so you must manually create audit log configuration and audit policies. By default, K3s enables both the `PodSecurity` and `NodeRestriction` admission controllers, among others.
<Tabs>
<TabItem value="v1.25 and Newer" default>
K3s v1.25 and newer support [Pod Security Admissions (PSAs)](https://kubernetes.io/docs/concepts/security/pod-security-admission/) for controlling pod security.
You can specify the Pod Security Admission configuration by setting the `defaultPodSecurityAdmissionConfigurationTemplateName` field in the cluster configuration in Rancher:
```yaml
spec:
defaultPodSecurityAdmissionConfigurationTemplateName: rancher-restricted
```
</TabItem>
<TabItem value="v1.24 and Older">
K3s v1.24 and older support [Pod Security Policies (PSPs)](https://v1-24.docs.kubernetes.io/docs/concepts/security/pod-security-policy/) for controlling pod security.
You can enable PSPs by passing the following flags in the cluster configuration in Rancher:
```yaml
spec:
rkeConfig:
machineGlobalConfig:
kube-apiserver-arg:
- enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount
```
This maintains the `NodeRestriction` plugin and enables the `PodSecurityPolicy`.
Once you enable PSPs, you can apply a policy to satisfy the necessary controls described in section 5.2 of the CIS Benchmark.
Here's an example of a compliant PSP:
```yaml
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
name: restricted-psp
spec:
privileged: false # CIS - 5.2.1
allowPrivilegeEscalation: false # CIS - 5.2.5
requiredDropCapabilities: # CIS - 5.2.7/8/9
- ALL
volumes:
- 'configMap'
- 'emptyDir'
- 'projected'
- 'secret'
- 'downwardAPI'
- 'csi'
- 'persistentVolumeClaim'
- 'ephemeral'
hostNetwork: false # CIS - 5.2.4
hostIPC: false # CIS - 5.2.3
hostPID: false # CIS - 5.2.2
runAsUser:
rule: 'MustRunAsNonRoot' # CIS - 5.2.6
seLinux:
rule: 'RunAsAny'
supplementalGroups:
rule: 'MustRunAs'
ranges:
- min: 1
max: 65535
fsGroup:
rule: 'MustRunAs'
ranges:
- min: 1
max: 65535
readOnlyRootFilesystem: false
```
For the example PSP to be effective, we need to create a `ClusterRole` and a `ClusterRoleBinding`. We also need to include a "system unrestricted policy" for system-level pods that require additional privileges, and an additional policy that allows the necessary sysctls for full functionality of ServiceLB.
If we combine the configuration above with the Network Policy described in the next section, we can place a single file in the `/var/lib/rancher/k3s/server/manifests` directory. Here is an example of a `policy.yaml` file:
```yaml
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
name: restricted-psp
spec:
privileged: false
allowPrivilegeEscalation: false
requiredDropCapabilities:
- ALL
volumes:
- 'configMap'
- 'emptyDir'
- 'projected'
- 'secret'
- 'downwardAPI'
- 'csi'
- 'persistentVolumeClaim'
- 'ephemeral'
hostNetwork: false
hostIPC: false
hostPID: false
runAsUser:
rule: 'MustRunAsNonRoot'
seLinux:
rule: 'RunAsAny'
supplementalGroups:
rule: 'MustRunAs'
ranges:
- min: 1
max: 65535
fsGroup:
rule: 'MustRunAs'
ranges:
- min: 1
max: 65535
readOnlyRootFilesystem: false
---
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
name: system-unrestricted-psp
annotations:
seccomp.security.alpha.kubernetes.io/allowedProfileNames: '*'
spec:
allowPrivilegeEscalation: true
allowedCapabilities:
- '*'
fsGroup:
rule: RunAsAny
hostIPC: true
hostNetwork: true
hostPID: true
hostPorts:
- max: 65535
min: 0
privileged: true
runAsUser:
rule: RunAsAny
seLinux:
rule: RunAsAny
supplementalGroups:
rule: RunAsAny
volumes:
- '*'
---
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
name: svclb-psp
annotations:
seccomp.security.alpha.kubernetes.io/allowedProfileNames: '*'
spec:
allowPrivilegeEscalation: false
allowedCapabilities:
- NET_ADMIN
allowedUnsafeSysctls:
- net.ipv4.ip_forward
- net.ipv6.conf.all.forwarding
fsGroup:
rule: RunAsAny
hostPorts:
- max: 65535
min: 0
runAsUser:
rule: RunAsAny
seLinux:
rule: RunAsAny
supplementalGroups:
rule: RunAsAny
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: psp:restricted-psp
rules:
- apiGroups:
- policy
resources:
- podsecuritypolicies
verbs:
- use
resourceNames:
- restricted-psp
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: psp:system-unrestricted-psp
rules:
- apiGroups:
- policy
resources:
- podsecuritypolicies
resourceNames:
- system-unrestricted-psp
verbs:
- use
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: psp:svclb-psp
rules:
- apiGroups:
- policy
resources:
- podsecuritypolicies
resourceNames:
- svclb-psp
verbs:
- use
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: default:restricted-psp
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: psp:restricted-psp
subjects:
- kind: Group
name: system:authenticated
apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: system-unrestricted-node-psp-rolebinding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: psp:system-unrestricted-psp
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: Group
name: system:nodes
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: system-unrestricted-svc-acct-psp-rolebinding
namespace: kube-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: psp:system-unrestricted-psp
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: Group
name: system:serviceaccounts
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: svclb-psp-rolebinding
namespace: kube-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: psp:svclb-psp
subjects:
- kind: ServiceAccount
name: svclb
---
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
name: intra-namespace
namespace: kube-system
spec:
podSelector: {}
ingress:
- from:
- namespaceSelector:
matchLabels:
name: kube-system
---
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
name: intra-namespace
namespace: default
spec:
podSelector: {}
ingress:
- from:
- namespaceSelector:
matchLabels:
name: default
---
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
name: intra-namespace
namespace: kube-public
spec:
podSelector: {}
ingress:
- from:
- namespaceSelector:
matchLabels:
name: kube-public
```
:::note
The critical Kubernetes additions such as CNI, DNS, and Ingress are run as pods in the `kube-system` namespace. Therefore, this namespace has a less restrictive policy, so that these components can run properly.
:::
</TabItem>
</Tabs>
### Network Policies
CIS requires that all namespaces apply a network policy that reasonably limits traffic into namespaces and pods.
:::note
This is a manual check in the CIS Benchmark. The CIS scan flags the result as a `warning`, because manual inspection is necessary by the cluster operator.
:::
Here's an example of a compliant network policy:
```yaml
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
name: intra-namespace
namespace: kube-system
spec:
podSelector: {}
ingress:
- from:
- namespaceSelector:
matchLabels:
name: kube-system
```
The active restrictions block DNS unless purposely allowed. Below is a network policy that allows DNS-related traffic:
```yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-network-dns-policy
namespace: <NAMESPACE>
spec:
ingress:
- ports:
- port: 53
protocol: TCP
- port: 53
protocol: UDP
podSelector:
matchLabels:
k8s-app: kube-dns
policyTypes:
- Ingress
```
The metrics-server and Traefik ingress controller are blocked by default if network policies aren't created to allow access.
<Tabs>
<TabItem value="v1.21 and Newer" default>
```yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-all-metrics-server
namespace: kube-system
spec:
podSelector:
matchLabels:
k8s-app: metrics-server
ingress:
- {}
policyTypes:
- Ingress
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-all-svclbtraefik-ingress
namespace: kube-system
spec:
podSelector:
matchLabels:
svccontroller.k3s.cattle.io/svcname: traefik
ingress:
- {}
policyTypes:
- Ingress
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-all-traefik-v121-ingress
namespace: kube-system
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: traefik
ingress:
- {}
policyTypes:
- Ingress
---
```
</TabItem>
<TabItem value="v1.20 and Older">
```yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-all-metrics-server
namespace: kube-system
spec:
podSelector:
matchLabels:
k8s-app: metrics-server
ingress:
- {}
policyTypes:
- Ingress
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-all-svclbtraefik-ingress
namespace: kube-system
spec:
podSelector:
matchLabels:
svccontroller.k3s.cattle.io/svcname: traefik
ingress:
- {}
policyTypes:
- Ingress
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-all-traefik-v120-ingress
namespace: kube-system
spec:
podSelector:
matchLabels:
app: traefik
ingress:
- {}
policyTypes:
- Ingress
---
```
</TabItem>
</Tabs>
:::note
You must manage network policies as normal for any additional namespaces you create.
:::
### API Server audit configuration
K3s doesn't create the log directory and audit policy by default, as auditing requirements are specific to each user's policies and environment.
If you need a log directory, it must be created before you start K3s. We recommend a restrictive access permission to avoid leaking sensitive information.
```bash
sudo mkdir -p -m 700 /var/lib/rancher/k3s/server/logs
```
The following is a starter audit policy to log request metadata. This policy should be written to a file named `audit.yaml` in the `/var/lib/rancher/k3s/server` directory. Detailed information about policy configuration for the API server can be found in the [official Kubernetes documentation](https://kubernetes.io/docs/tasks/debug/debug-cluster/audit/).
```yaml
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
- level: Metadata
```
Further configurations are also needed to pass CIS checks. These aren't configured by default in K3s, because they vary based on your environment and needs:
- Ensure that the `--audit-log-path` argument is set.
- Ensure that the `--audit-log-maxage` argument is set to 30 or as appropriate.
- Ensure that the `--audit-log-maxbackup` argument is set to 10 or as appropriate.
- Ensure that the `--audit-log-maxsize` argument is set to 100 or as appropriate.
Combined, to enable and configure audit logs, add the following lines to the K3s cluster configuration file in Rancher:
```yaml
spec:
rkeConfig:
machineGlobalConfig:
kube-apiserver-arg:
- audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml # CIS v1.20/v1.23 3.2.1
- audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log # CIS v1.20 1.2.21 and CIS v1.23 1.2.19
- audit-log-maxage=30 # CIS v1.20 1.2.22 and CIS v1.23 1.2.20
- audit-log-maxbackup=10 # CIS v1.20 1.2.23 and CIS v1.23 1.2.21
- audit-log-maxsize=100 # CIS v1.20 1.2.24 and CIS v1.23 1.2.22
```
## Reference Hardened K3s Template Configuration
The following reference template configuration is used in Rancher to create a hardened K3s custom cluster based on each CIS control in this guide. This reference doesn't include other required **cluster configuration** directives, which vary based on your environment.
<Tabs groupId="rke2-version">
<TabItem value='v1.25 and Newer' default>
```yaml
apiVersion: provisioning.cattle.io/v1
kind: Cluster
metadata:
name: # Define cluster name
spec:
defaultPodSecurityAdmissionConfigurationTemplateName: rancher-restricted
enableNetworkPolicy: true
kubernetesVersion: # Define K3s version
rkeConfig:
machineGlobalConfig:
kube-apiserver-arg:
- enable-admission-plugins=NodeRestriction,ServiceAccount # CIS 1.2.16, CIS 5.2 and CIS v1.20 - 1.2.13 / CIS v1.23 - 1.2.14
- audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml # CIS v1.20/v1.23 3.2.1
- audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log # CIS v1.20 1.2.21 and CIS v1.23 1.2.19
- audit-log-maxage=30 # CIS v1.20 1.2.22 and CIS v1.23 1.2.20
- audit-log-maxbackup=10 # CIS v1.20 1.2.23 and CIS v1.23 1.2.21
- audit-log-maxsize=100 # CIS v1.20 1.2.24 and CIS v1.23 1.2.22
- request-timeout=300s # Control CIS v1.20 1.2.26 and CIS v1.23 1.2.23
- service-account-lookup=true # Control CIS v1.23 1.2.24
secrets-encryption: true
machineSelectorConfig:
- config:
kubelet-arg:
- make-iptables-util-chains=true # Control 4.2.7
protect-kernel-defaults: true # Control 4.2.6
```
</TabItem>
<TabItem value='v1.22 up to 1.24'>
```yaml
apiVersion: provisioning.cattle.io/v1
kind: Cluster
metadata:
name: # Define cluster name
spec:
enableNetworkPolicy: true
kubernetesVersion: # Define K3s version
rkeConfig:
machineGlobalConfig:
kube-apiserver-arg:
- enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount # CIS 1.2.16, CIS 5.2 and CIS v1.20 - 1.2.13 / CIS v1.23 - 1.2.14
- audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml # CIS v1.20/v1.23 3.2.1
- audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log # CIS v1.20 1.2.21 and CIS v1.23 1.2.19
- audit-log-maxage=30 # CIS v1.20 1.2.22 and CIS v1.23 1.2.20
- audit-log-maxbackup=10 # CIS v1.20 1.2.23 and CIS v1.23 1.2.21
- audit-log-maxsize=100 # CIS v1.20 1.2.24 and CIS v1.23 1.2.22
- request-timeout=300s # Control CIS v1.20 1.2.26 and CIS v1.23 1.2.23
- service-account-lookup=true # Control CIS v1.23 1.2.24
secrets-encryption: true
machineSelectorConfig:
- config:
kubelet-arg:
- make-iptables-util-chains=true # Control 4.2.7
protect-kernel-defaults: true # Control 4.2.6
```
</TabItem>
</Tabs>
## Conclusion
If you've followed this guide, your K3s custom cluster provisioned by Rancher will be configured to pass the CIS Kubernetes Benchmark. You can review our K3s self-assessment guides to understand how we verified each of the benchmarks and how you can do the same on your cluster.
docs/reference-guides/rancher-security/hardening-guides/k3s-self-assessment-guide-with-cis-v1.23-k8s-v1.23.md
+3148
View File
File diff suppressed because it is too large Load Diff
docs/reference-guides/rancher-security/hardening-guides/k3s-self-assessment-guide-with-cis-v1.23-k8s-v1.24.md
+3148
View File
File diff suppressed because it is too large Load Diff
docs/reference-guides/rancher-security/hardening-guides/k3s-self-assessment-guide-with-cis-v1.23-k8s-v1.25.md
+3148
View File
File diff suppressed because it is too large Load Diff
docs/reference-guides/rancher-security/rancher-v2.6-hardening-guides/rke1-hardening-guide-with-cis-v1.6-benchmark.md → docs/reference-guides/rancher-security/hardening-guides/rke1-hardening-guide.md
+288 -207
View File
@@ -1,34 +1,22 @@
---
title: RKE Hardening Guide with CIS v1.6 Benchmark
title: RKE Hardening Guide
---
This document provides prescriptive guidance for hardening a production installation of a RKE cluster to be used with Rancher v2.6. It outlines the configurations and controls required to address Kubernetes benchmark controls from the Center for Information Security (CIS).
This document provides prescriptive guidance for how to harden an RKE cluster intended for production, before provisioning it with Rancher. It outlines the configurations and controls required for Center for Information Security (CIS) Kubernetes benchmark controls.
:::note
This hardening guide describes how to secure the nodes in your cluster, and it is recommended to follow this guide before installing Kubernetes.
This hardening guide describes how to secure the nodes in your cluster. We recommended that you follow this guide before you install Kubernetes.
:::
This hardening guide is intended to be used for RKE clusters and associated with specific versions of the CIS Kubernetes Benchmark, Kubernetes, and Rancher:
This hardening guide is intended to be used for RKE clusters and associated with the following versions of the CIS Kubernetes Benchmark, Kubernetes, and Rancher:
| Rancher Version | CIS Benchmark Version | Kubernetes Version |
| --------------- | --------------------- | ------------------ |
| Rancher v2.6 | Benchmark v1.6 | Kubernetes v1.18 up to v1.23 |
| Rancher Version | CIS Benchmark Version | Kubernetes Version |
|-----------------|-----------------------|------------------------------|
| Rancher v2.7 | Benchmark v1.23 | Kubernetes v1.23 up to v1.25 |
[Click here to download a PDF version of this document](https://releases.rancher.com/documents/security/2.6/Rancher_v2-6_CIS_v1-6_Hardening_Guide.pdf).
For more details on how to evaluate a hardened RKE cluster against the official CIS benchmark, refer to the RKE self-assessment guides for specific CIS benchmark versions.
### Overview
This document provides prescriptive guidance for hardening a RKE cluster to be used for installing Rancher v2.6 with Kubernetes v1.18 up to v1.23 or provisioning a RKE cluster with Kubernetes v1.18 up to v1.23 to be used within Rancher v2.6. It outlines the configurations required to address Kubernetes benchmark controls from the Center for Information Security (CIS).
For more details about evaluating a hardened cluster against the official CIS benchmark, refer to the [CIS 1.6 Benchmark - Self-Assessment Guide - Rancher v2.6](./rke1-self-assessment-guide-with-cis-v1.6-benchmark.md).
#### Known Issues
- Rancher **exec shell** and **view logs** for pods are **not** functional in a CIS v1.6 hardened setup when only public IP is provided when registering custom nodes. This functionality requires a private IP to be provided when registering the custom nodes.
- When setting the `default_pod_security_policy_template_id:` to `restricted` or `restricted-noroot`, based on the pod security policies (PSP) [provided](../../../how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/create-pod-security-policies.md) by Rancher, Rancher creates **RoleBindings** and **ClusterRoleBindings** on the default service accounts. The CIS v1.6 check 5.1.5 requires that the default service accounts have no roles or cluster roles bound to it apart from the defaults. In addition the default service accounts should be configured such that it does not provide a service account token and does not have any explicit rights assignments.
## Host-level requirements
### Configure Kernel Runtime Parameters
@@ -46,11 +34,14 @@ Run `sysctl -p /etc/sysctl.d/90-kubelet.conf` to enable the settings.
### Configure `etcd` user and group
A user account and group for the **etcd** service is required to be setup before installing RKE. The **uid** and **gid** for the **etcd** user will be used in the RKE **config.yml** to set the proper permissions for files and directories during installation time.
A user account and group for the **etcd** service is required to be setup before installing RKE.
The **uid** and **gid** for the **etcd** user will be used in the RKE **config.yml** to set the proper permissions for files and directories during installation time.
#### Create `etcd` user and group
To create the **etcd** user and group run the following console commands. The commands below use `52034` for **uid** and **gid** are for example purposes. Any valid unused **uid** or **gid** could also be used in lieu of `52034`.
To create the **etcd** user and group run the following console commands.
The commands below use `52034` for **uid** and **gid** for example purposes.
Any valid unused **uid** or **gid** could also be used in lieu of `52034`.
```bash
groupadd --gid 52034 etcd
@@ -66,11 +57,15 @@ services:
uid: 52034
```
## Kubernetes runtime requirements
### Configure `default` Service Account
#### Set `automountServiceAccountToken` to `false` for `default` service accounts
Kubernetes provides a default service account which is used by cluster workloads where no specific service account is assigned to the pod. Where access to the Kubernetes API from a pod is required, a specific service account should be created for that pod, and rights granted to that service account. The default service account should be configured such that it does not provide a service account token and does not have any explicit rights assignments.
Kubernetes provides a default service account which is used by cluster workloads where no specific service account is assigned to the pod.
Where access to the Kubernetes API from a pod is required, a specific service account should be created for that pod, and rights granted to that service account.
The default service account should be configured such that it does not provide a service account token and does not have any explicit rights assignments.
For each namespace including **default** and **kube-system** on a standard RKE install, the **default** service account must include this value:
@@ -88,7 +83,8 @@ metadata:
automountServiceAccountToken: false
```
Create a bash script file called `account_update.sh`. Be sure to `chmod +x account_update.sh` so the script has execute permissions.
Create a bash script file called `account_update.sh`.
Be sure to `chmod +x account_update.sh` so the script has execute permissions.
```bash
#!/bin/bash -e
@@ -109,9 +105,7 @@ Network Policies are namespace scoped. When a network policy is introduced to a
Once a CNI provider is enabled on a cluster a default network policy can be applied. For reference purposes a **permissive** example is provided below. If you want to allow all traffic to all pods in a namespace (even if policies are added that cause some pods to be treated as “isolated”), you can create a policy that explicitly allows all traffic in that namespace. Save the following configuration as `default-allow-all.yaml`. Additional [documentation](https://kubernetes.io/docs/concepts/services-networking/network-policies/) about network policies can be found on the Kubernetes site.
:::note
This `NetworkPolicy` is just an example and is not recommended for production use.
:::
```yaml
@@ -143,124 +137,149 @@ done
Execute this script to apply the `default-allow-all.yaml` configuration with the **permissive** `NetworkPolicy` to all namespaces.
### Reference Hardened RKE `cluster.yml` Configuration
The reference `cluster.yml` is used by the RKE CLI that provides the configuration needed to achieve a hardened install of Rancher Kubernetes Engine (RKE). RKE install [documentation](https://rancher.com/docs/rke/latest/en/installation/) is provided with additional details about the configuration items. This reference `cluster.yml` does not include the required **nodes** directive which will vary depending on your environment. Documentation for node configuration in RKE can be found [here](https://rancher.com/docs/rke/latest/en/config-options/nodes/).
:::note Important:
## Known Limitations
For a Kubernetes v1.18 cluster, the configuration `spec.volumes: 'ephemeral'` should be removed from the `PodSecurityPolicy`, since it's not supported in this Kubernetes release.
- Rancher **exec shell** and **view logs** for pods are **not** functional in a hardened setup when only a public IP is provided when registering custom nodes. This functionality requires a private IP to be provided when registering the custom nodes.
- When setting `default_pod_security_policy_template_id:` to `restricted` or `restricted-noroot`, based on the pod security policies (PSP) [provided](../../../how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/create-pod-security-policies.md) by Rancher, Rancher creates **RoleBindings** and **ClusterRoleBindings** on the default service accounts. The CIS v1.6/v1.20/v1.23 check 5.1.5 requires that the default service accounts have no roles or cluster roles bound to it apart from the defaults. In addition, the default service accounts should be configured such that it does not provide a service account token and does not have any explicit rights assignments.
## Known Issues
The following are controls that RKE currently does not pass. Each gap will be explained and whether it can be passed through manual operator intervention or if it will be addressed in a future release.
<Tabs groupId="rke-version">
<TabItem value='v1.25 and Newer' default>
#### CIS v1.23 - Control 5.2.3 - Minimize the admission of containers wishing to share the host process ID namespace (Automated)
#### CIS v1.23 - Control 5.2.4 - Minimize the admission of containers wishing to share the host IPC namespace (Automated)
#### CIS v1.23 - Control 5.2.5 - Minimize the admission of containers wishing to share the host network namespace (Automated)
#### CIS v1.23 - Control 5.2.6 - Minimize the admission of containers with allowPrivilegeEscalation (Automated)
**Rationale**
Those controls are defined to evaluate certain PodSecurityPolicy resources in the benchmark, but the PodSecurityPolicy API is removed starting in Kubernetes v1.25, therefore those controls fail in the scan.
**Remediation**
Pod Security Admission is configured by setting the Pod Security Admission Configuration Template (PSACT) `rancher-restricted` in the cluster to enforce the restriction on applicable namespaces.
Due to the lack of utilizing PSA in the current version of the benchmark, the above controls will not be addressed until an updated version of the benchmark is available.
#### CIS v1.23 - Control 5.3.2 - Ensure that all Namespaces have Network Policies defined (Automated)
**Rationale**
The benchmark scanner expects to see one or more `networkpolicy` resources in every namespace.
**Remediation**
This is a known issue that rancher-cis-benchmark fails on validating this control. See [issue](https://github.com/rancher/rancher/issues/40648).
</TabItem>
<TabItem value='v1.22 up to v1.24'>
#### CIS v1.23 - Control 5.3.2 - Ensure that all Namespaces have Network Policies defined (Automated)
**Rationale**
The benchmark scanner expects to see one or more `networkpolicy` resources in every namespace.
**Remediation**
This is a known issue that rancher-cis-benchmark fails on validating this control. See [issue](https://github.com/rancher/rancher/issues/40648).
</TabItem>
</Tabs>
## Reference Hardened RKE `cluster.yml` Configuration
The reference `cluster.yml` is used by the RKE CLI that provides the configuration needed to achieve a hardened installation of Rancher Kubernetes Engine (RKE). RKE [documentation](https://rancher.com/docs/rke/latest/en/installation/) provides additional details about the configuration items. This reference `cluster.yml` does not include the required **nodes** directive which will vary depending on your environment. Documentation for node configuration in RKE can be found [here](https://rancher.com/docs/rke/latest/en/config-options/nodes/).
<Tabs groupId="rke-version">
<TabItem value='v1.25 and Newer' default>
:::note
If you intend to import an RKE cluster into Rancher, please consult the documentation for how to configure the Pod Security Admission to add Rancher's namespaces to the exemption list.
:::
```yaml
# If you intend to deploy Kubernetes in an air-gapped environment,
# please consult the documentation on how to configure custom RKE images.
# https://rancher.com/docs/rke/latest/en/installation/ .
# The nodes directive is required and will vary depending on your environment.
# Documentation for node configuration can be found here:
# https://rancher.com/docs/rke/latest/en/config-options/nodes/
nodes: []
kubernetes_version: v1.25.6-rancher2-1
services:
etcd:
image: ""
extra_args: {}
extra_binds: []
extra_env: []
win_extra_args: {}
win_extra_binds: []
win_extra_env: []
external_urls: []
ca_cert: ""
cert: ""
key: ""
path: ""
uid: 52034
gid: 52034
snapshot: false
retention: ""
creation: ""
backup_config: null
kube-api:
image: ""
extra_args: {}
extra_binds: []
extra_env: []
win_extra_args: {}
win_extra_binds: []
win_extra_env: []
service_cluster_ip_range: ""
service_node_port_range: ""
pod_security_policy: true
always_pull_images: false
secrets_encryption_config:
enabled: true
custom_config: null
audit_log:
enabled: true
configuration: null
admission_configuration: null
event_rate_limit:
enabled: true
configuration: null
pod_security_configuration: restricted
kube-controller:
extra_args:
feature-gates: RotateKubeletServerCertificate=true
kubelet:
extra_args:
feature-gates: RotateKubeletServerCertificate=true
protect-kernel-defaults: "true"
generate_serving_certificate: true
addons: |
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-allow-all
spec:
podSelector: {}
ingress:
- {}
egress:
- {}
policyTypes:
- Ingress
- Egress
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: default
automountServiceAccountToken: false
```
</TabItem>
<TabItem value='v1.22 up to v1.24'>
```yaml
# If you intend to deploy Kubernetes in an air-gapped environment,
# please consult the documentation on how to configure custom RKE images.
nodes: []
kubernetes_version: v1.24.10-rancher3-1
services:
etcd:
uid: 52034
gid: 52034
kube-api:
secrets_encryption_config:
enabled: true
audit_log:
enabled: true
event_rate_limit:
enabled: true
pod_security_policy: true
kube-controller:
image: ""
extra_args:
feature-gates: RotateKubeletServerCertificate=true
tls-cipher-suites: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256
bind-address: 127.0.0.1
extra_binds: []
extra_env: []
win_extra_args: {}
win_extra_binds: []
win_extra_env: []
cluster_cidr: ""
service_cluster_ip_range: ""
scheduler:
image: ""
extra_args:
tls-cipher-suites: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256
bind-address: 127.0.0.1
extra_binds: []
extra_env: []
win_extra_args: {}
win_extra_binds: []
win_extra_env: []
kubelet:
image: ""
extra_args:
feature-gates: RotateKubeletServerCertificate=true
protect-kernel-defaults: true
tls-cipher-suites: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256
extra_binds: []
extra_env: []
win_extra_args: {}
win_extra_binds: []
win_extra_env: []
cluster_domain: cluster.local
infra_container_image: ""
cluster_dns_server: ""
fail_swap_on: false
generate_serving_certificate: true
kubeproxy:
image: ""
extra_args: {}
extra_binds: []
extra_env: []
win_extra_args: {}
win_extra_binds: []
win_extra_env: []
network:
plugin: ""
options: {}
mtu: 0
node_selector: {}
update_strategy: null
authentication:
strategy: ""
sans: []
webhook: null
addons: |
# Upstream Kubernetes restricted PSP policy
# https://github.com/kubernetes/website/blob/564baf15c102412522e9c8fc6ef2b5ff5b6e766c/content/en/examples/policy/restricted-psp.yaml
@@ -357,96 +376,152 @@ addons: |
metadata:
name: default
automountServiceAccountToken: false
addons_include: []
system_images:
etcd: ""
alpine: ""
nginx_proxy: ""
cert_downloader: ""
kubernetes_services_sidecar: ""
kubedns: ""
dnsmasq: ""
kubedns_sidecar: ""
kubedns_autoscaler: ""
coredns: ""
coredns_autoscaler: ""
nodelocal: ""
kubernetes: ""
flannel: ""
flannel_cni: ""
calico_node: ""
calico_cni: ""
calico_controllers: ""
calico_ctl: ""
calico_flexvol: ""
canal_node: ""
canal_cni: ""
canal_controllers: ""
canal_flannel: ""
canal_flexvol: ""
weave_node: ""
weave_cni: ""
pod_infra_container: ""
ingress: ""
ingress_backend: ""
metrics_server: ""
windows_pod_infra_container: ""
ssh_key_path: ""
ssh_cert_path: ""
ssh_agent_auth: false
authorization:
mode: ""
options: {}
ignore_docker_version: false
kubernetes_version: ""
private_registries: []
ingress:
provider: ""
options: {}
node_selector: {}
extra_args: {}
dns_policy: ""
extra_envs: []
extra_volumes: []
extra_volume_mounts: []
update_strategy: null
http_port: 0
https_port: 0
network_mode: ""
cluster_name:
cloud_provider:
name: ""
prefix_path: ""
win_prefix_path: ""
addon_job_timeout: 0
bastion_host:
address: ""
port: ""
user: ""
ssh_key: ""
ssh_key_path: ""
ssh_cert: ""
ssh_cert_path: ""
monitoring:
provider: ""
options: {}
node_selector: {}
update_strategy: null
replicas: null
restore:
restore: false
snapshot_name: ""
dns: null
upgrade_strategy:
max_unavailable_worker: ""
max_unavailable_controlplane: ""
drain: null
node_drain_input: null
```
### Reference Hardened RKE Template Configuration
</TabItem>
</Tabs>
The reference RKE template provides the configuration needed to achieve a hardened install of Kubernetes. RKE templates are used to provision Kubernetes and define Rancher settings. Follow the Rancher [documentation](../../../pages-for-subheaders/installation-and-upgrade.md) for additional installation and RKE template details.
## Reference Hardened RKE Template Configuration
The reference RKE template provides the configuration needed to achieve a hardened installation of Kubernetes. RKE templates are used to provision Kubernetes and define Rancher settings. Follow the Rancher [documentation](../../../pages-for-subheaders/installation-and-upgrade.md) for additional installation and RKE template details.
<Tabs groupId="rke-version">
<TabItem value='v1.25 and Newer' default>
```yaml
#
# Cluster Config
#
default_pod_security_admission_configuration_template_name: rancher-restricted
docker_root_dir: /var/lib/docker
enable_cluster_alerting: false
enable_cluster_monitoring: false
enable_network_policy: true
local_cluster_auth_endpoint:
enabled: true
name: ''
#
# Rancher Config
#
rancher_kubernetes_engine_config:
addon_job_timeout: 45
authentication:
strategy: x509
dns:
nodelocal:
ip_address: ''
node_selector: null
update_strategy: {}
enable_cri_dockerd: false
ignore_docker_version: true
#
# # Currently only nginx ingress provider is supported.
# # To disable ingress controller, set `provider: none`
# # To enable ingress on specific nodes, use the node_selector, eg:
# provider: nginx
# node_selector:
# app: ingress
#
ingress:
default_backend: false
default_ingress_class: true
http_port: 0
https_port: 0
provider: nginx
kubernetes_version: v1.25.6-rancher2-1
monitoring:
provider: metrics-server
replicas: 1
#
# If you are using calico on AWS
#
# network:
# plugin: calico
# calico_network_provider:
# cloud_provider: aws
#
# # To specify flannel interface
#
# network:
# plugin: flannel
# flannel_network_provider:
# iface: eth1
#
# # To specify flannel interface for canal plugin
#
# network:
# plugin: canal
# canal_network_provider:
# iface: eth1
#
network:
mtu: 0
options:
flannel_backend_type: vxlan
plugin: canal
rotate_encryption_key: false
#
# services:
# kube-api:
# service_cluster_ip_range: 10.43.0.0/16
# kube-controller:
# cluster_cidr: 10.42.0.0/16
# service_cluster_ip_range: 10.43.0.0/16
# kubelet:
# cluster_domain: cluster.local
# cluster_dns_server: 10.43.0.10
#
services:
scheduler:
extra_args:
tls-cipher-suites: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256
bind-address: 127.0.0.1
etcd:
backup_config:
enabled: true
interval_hours: 12
retention: 6
safe_timestamp: false
timeout: 300
creation: 12h
extra_args:
election-timeout: 5000
heartbeat-interval: 500
retention: 72h
snapshot: false
uid: 52034
gid: 52034
kube_api:
always_pull_images: false
audit_log:
enabled: true
event_rate_limit:
enabled: true
pod_security_policy: false
secrets_encryption_config:
enabled: true
service_node_port_range: 30000-32767
kube-controller:
extra_args:
feature-gates: RotateKubeletServerCertificate=true
tls-cipher-suites: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256
bind-address: 127.0.0.1
kubelet:
extra_args:
feature-gates: RotateKubeletServerCertificate=true
protect-kernel-defaults: true
tls-cipher-suites: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256
fail_swap_on: false
generate_serving_certificate: true
ssh_agent_auth: false
upgrade_strategy:
max_unavailable_controlplane: '1'
max_unavailable_worker: 10%
windows_prefered_cluster: false
```
</TabItem>
<TabItem value='v1.22 up to v1.24'>
```yaml
#
@@ -488,7 +563,7 @@ rancher_kubernetes_engine_config:
http_port: 0
https_port: 0
provider: nginx
kubernetes_version: v1.21.8-rancher1-1
kubernetes_version: v1.24.10-rancher3-1
monitoring:
provider: metrics-server
replicas: 1
@@ -580,11 +655,14 @@ rancher_kubernetes_engine_config:
windows_prefered_cluster: false
```
### Reference Hardened **cloud-config** Configuration
</TabItem>
</Tabs>
## Reference Hardened **cloud-config** Configuration
A **cloud-config** configuration file is generally used in cloud infrastructure environments to allow for configuration management of compute instances. The reference config configures SUSE Linux Enterprise Server (SLES), openSUSE Leap, Red Hat Enterprise Linux (RHEL) and Ubuntu operating system level settings needed before installing Kubernetes.
#### Reference Hardened **cloud-config** for SUSE Linux Enterprise Server 15 (SLES 15) and openSUSE Leap 15
### Reference Hardened **cloud-config** for SUSE Linux Enterprise Server 15 (SLES 15) and openSUSE Leap 15
```yaml
#cloud-config
@@ -615,7 +693,7 @@ runcmd:
- useradd --comment "etcd service account" --uid 52034 --gid 52034 etcd --shell /usr/sbin/nologin
```
#### Reference Hardened **cloud-config** for Red Hat Enterprise Linux 8 (RHEL 8) and Ubuntu 20.04 LTS
### Reference Hardened **cloud-config** for Red Hat Enterprise Linux 8 (RHEL 8) and Ubuntu 20.04 LTS
```yaml
#cloud-config
@@ -642,3 +720,6 @@ runcmd:
- groupadd --gid 52034 etcd
- useradd --comment "etcd service account" --uid 52034 --gid 52034 etcd --shell /usr/sbin/nologin
```
## Conclusion
If you have followed this guide, your RKE custom cluster provisioned by Rancher will be configured to pass the CIS Kubernetes Benchmark. You can review our RKE self-assessment guides to understand how we verified each of the benchmarks and how you can do the same on your cluster.
docs/reference-guides/rancher-security/hardening-guides/rke1-self-assessment-guide-with-cis-v1.23-k8s-v1.23.md
+3083
View File
File diff suppressed because one or more lines are too long
docs/reference-guides/rancher-security/hardening-guides/rke1-self-assessment-guide-with-cis-v1.23-k8s-v1.24.md
+3084
View File
File diff suppressed because one or more lines are too long
docs/reference-guides/rancher-security/hardening-guides/rke1-self-assessment-guide-with-cis-v1.23-k8s-v1.25.md
+3085
View File
File diff suppressed because one or more lines are too long
docs/reference-guides/rancher-security/hardening-guides/rke2-hardening-guide.md
+353
View File
@@ -0,0 +1,353 @@
---
title: RKE2 Hardening Guide
---
This document provides prescriptive guidance for how to harden an RKE2 cluster intended for production, before provisioning it with Rancher. It outlines the configurations and controls required for Center for Information Security (CIS) Kubernetes benchmark controls.
:::note
This hardening guide describes how to secure the nodes in your cluster. We recommended that you follow this guide before you install Kubernetes.
:::
This hardening guide is intended to be used for RKE2 clusters and associated with the following versions of the CIS Kubernetes Benchmark, Kubernetes, and Rancher:
| Rancher Version | CIS Benchmark Version | Kubernetes Version |
|-----------------|-----------------------|------------------------------|
| Rancher v2.7 | Benchmark v1.23 | Kubernetes v1.22 up to v1.25 |
For more details on how to evaluate a hardened RKE2 cluster against the official CIS benchmark, refer to the RKE2 self-assessment guides for specific CIS benchmark versions.
RKE2 passes a number of the Kubernetes CIS controls without modification, as it applies several security mitigations by default. There are some notable exceptions to this that require manual intervention to fully comply with the CIS Benchmark:
1. RKE2 will not modify the host operating system. Therefore, you, the operator, must make a few host-level modifications.
2. Certain CIS controls for Network Policies and Pod Security Standards (or Pod Security Policies (PSP) on RKE2 versions prior to v1.25) will restrict the functionality of the cluster. You must opt into having RKE2 configure these for you. To help ensure these requirements are met, RKE2 can be started with the profile flag set to `cis-1.5`, `cis-1.6`, or `cis-1.23`.
## Host-level requirements
There are two areas of host-level requirements: kernel parameters and etcd process/directory configuration. These are outlined in this section.
### Set kernel parameters
The following `sysctl` configuration is recommended for all nodes type in the cluster. Set the following parameters in `/etc/sysctl.d/90-kubelet.conf`:
```ini
vm.panic_on_oom=0
vm.overcommit_memory=1
kernel.panic=10
kernel.panic_on_oops=1
```
Run `sudo sysctl -p /etc/sysctl.d/90-kubelet.conf` to enable the settings.
### Ensure etcd is configured properly
The CIS Benchmark requires that the etcd data directory be owned by the `etcd` user and group. This implicitly requires the etcd process run as the host-level `etcd` user. To achieve this, RKE2 takes several steps when started with a valid `cis-1.xx` profile:
1. Check that the `etcd` user and group exists on the host. If they don't, exit with an error.
2. Create etcd's data directory with `etcd` as the user and group owner.
3. Ensure the etcd process is ran as the `etcd` user and group by setting the etcd static pod's `SecurityContext` appropriately.
To meet the above requirements, you must:
#### Create the etcd user
On some Linux distributions, the `useradd` command will not create a group. The `-U` flag is included below to account for that. This flag tells `useradd` to create a group with the same name as the user.
```bash
sudo useradd -r -c "etcd user" -s /sbin/nologin -M etcd -U
```
## Kubernetes runtime requirements
### Ensure `protect-kernel-defaults` is set
This is a kubelet flag that will cause the kubelet to exit if the required kernel parameters are unset or are set to values that are different from the kubelet's defaults.
Both `protect-kernel-defaults` and `profile` flags can be set in the RKE2 template configuration file.
When the `profile` flag is set, RKE2 will set the flag to `true` if it is unset.
```yaml
spec:
rkeConfig:
machineSelectorConfig:
- config:
profile: cis-1.23
protect-kernel-defaults: true
```
The runtime requirements to pass the CIS Benchmark are centered around pod security and network policies. Most of this is automatically handled by RKE2 when using a valid `cis-1.XX` profile, but some additional operator intervention is required. These are outlined in this section.
### PodSecurity
RKE2 always runs with some amount of pod security.
<Tabs groupId="rke2-version">
<TabItem value='v1.25 and Newer' default>
On v1.25 and newer, [Pod Security Admission (PSA)](https://kubernetes.io/docs/concepts/security/pod-security-admission/) are used for pod security.
Below is the minimum necessary configuration needed for hardening RKE2 to pass CIS v1.23 hardened profile `rke2-cis-1.23-profile-hardened` available in Rancher.
```yaml
spec:
defaultPodSecurityAdmissionConfigurationTemplateName: rancher-restricted
rkeConfig:
machineSelectorConfig:
- config:
profile: cis-1.23
protect-kernel-defaults: true
```
When both the `defaultPodSecurityAdmissionConfigurationTemplateName` and `profile` flags are set, Rancher and RKE2 does the following:
1. Checks that host-level requirements have been met. If they haven't, RKE2 will exit with a fatal error describing the unmet requirements.
2. Applies network policies that allow the cluster to pass associated controls.
3. Configures the Pod Security Admission Controller with the Pod Security Admission Configuration Template (PSACT) `rancher-restricted`, to enforce restricted mode in all namespaces, except the ones in the PSACT's exemption list.
These namespaces are exempted to allow system pods to run without restrictions, which is required for proper operation of the cluster.
</TabItem>
<TabItem value='v1.22 up to v1.24'>
On Kubernetes v1.22 to v1.24, the `PodSecurityPolicy` admission controller is always enabled.
Below is the minimum necessary configuration needed for hardening RKE2 to pass CIS v1.23 hardened profile `rke2-cis-1.23-profile-hardened` available in Rancher.
:::note
In the following example the profile is set to `cis-1.6` which is the value defined in the upstream RKE2, but the cluster is actually configured to pass the CIS v1.23 hardened profile
:::
```yaml
spec:
defaultPodSecurityPolicyTemplateName: restricted-noroot
rkeConfig:
machineSelectorConfig:
- config:
profile: cis-1.6
protect-kernel-defaults: true
```
When both the `defaultPodSecurityPolicyTemplateName` and `profile` flags are set, Rancher and RKE2 does the following:
1. Checks that host-level requirements have been met. If they haven't, RKE2 will exit with a fatal error describing the unmet requirements.
2. Applies network policies that allow the cluster to pass associated controls.
3. Configures runtime pod security policies that allow the cluster to pass associated controls.
</TabItem>
</Tabs>
:::note
The Kubernetes control plane components and critical additions such as CNI, DNS, and Ingress are ran as pods in the `kube-system` namespace. Therefore, this namespace will have a policy that is less restrictive so that these components can run properly.
:::
### NetworkPolicies
When ran with a valid `cis-1.XX` profile, RKE2 will put `NetworkPolicies` in place that passes the CIS Benchmark for Kubernetes' built-in namespaces. These namespaces are: `kube-system`, `kube-public`, `kube-node-lease`, and `default`.
The `NetworkPolicy` used will only allow pods within the same namespace to talk to each other. The notable exception to this is that it allows DNS requests to be resolved.
:::caution Operator Intervention Required
Operators must manage network policies as normal for additional namespaces that are created.
:::
### Configure `default` service account
**Set `automountServiceAccountToken` to `false` for `default` service accounts**
Kubernetes provides a `default` service account which is used by cluster workloads where no specific service account is assigned to the pod. Where access to the Kubernetes API from a pod is required, a specific service account should be created for that pod, and rights granted to that service account. The `default` service account should be configured such that it does not provide a service account token and does not have any explicit rights assignments.
For each namespace including `default` and `kube-system` on a standard RKE2 install, the `default` service account must include this value:
```yaml
automountServiceAccountToken: false
```
For namespaces created by the cluster operator, the following script and configuration file can be used to configure the `default` service account.
The configuration bellow must be saved to a file called `account_update.yaml`.
```yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: default
automountServiceAccountToken: false
```
Create a bash script file called `account_update.sh`. Be sure to `sudo chmod +x account_update.sh` so the script has execute permissions.
```bash
#!/bin/bash -e
for namespace in $(kubectl get namespaces -A -o=jsonpath="{.items[*]['metadata.name']}"); do
echo -n "Patching namespace $namespace - "
kubectl patch serviceaccount default -n ${namespace} -p "$(cat account_update.yaml)"
done
```
Execute this script to apply the `account_update.yaml` configuration to `default` service account in all namespaces.
### API Server audit configuration
CIS requirements 1.2.19 to 1.2.22 are related to configuring audit logs for the API Server. When RKE2 is started with the `profile` flag set, it will automatically configure hardened `--audit-log-` parameters in the API Server to pass those CIS checks.
RKE2's default audit policy is configured to not log requests in the API Server. This is done to allow cluster operators flexibility to customize an audit policy that suits their auditing requirements and needs, as these are specific to each users' environment and policies.
A default audit policy is created by RKE2 when started with the `profile` flag set. The policy is defined in `/etc/rancher/rke2/audit-policy.yaml`.
```yaml
apiVersion: audit.k8s.io/v1
kind: Policy
metadata:
creationTimestamp: null
rules:
- level: None
```
## Known issues
The following are controls that RKE2 currently does not pass. Each gap will be explained and whether it can be passed through manual operator intervention or if it will be addressed in a future release.
<Tabs groupId="rke2-version">
<TabItem value='v1.25 and Newer' default>
#### CIS v1.23 - Control 5.2.2 - Minimize the admission of privileged containers (Manual)
#### CIS v1.23 - Control 5.2.8 - Minimize the admission of containers with the NET_RAW capability (Automated)
**Rationale**
The benchmark scanner expects to see a PodSecurityPolicy resource named `global-restricted-psp` in the cluster. However, the PodSecurityPolicy resource Rancher creates is named `restricted-noroot-psp`.
**Remediation**
Create a PodSecurityPolicy resource named `global-restricted-psp` by cloning `restricted-noroot-psp`.
</TabItem>
<TabItem value='v1.22 up to v1.24'>
#### CIS v1.23 - Control 5.2.2 - Minimize the admission of privileged containers (Manual)
#### CIS v1.23 - Control 5.2.3 - Minimize the admission of containers wishing to share the host process ID namespace (Automated)
#### CIS v1.23 - Control 5.2.5 - Minimize the admission of containers wishing to share the host network namespace (Automated)
#### CIS v1.23 - Control 5.2.6 - Minimize the admission of containers with allowPrivilegeEscalation (Automated)
#### CIS v1.23 - Control 5.2.7 - Minimize the admission of root containers (Automated)
#### CIS v1.23 - Control 5.2.8 - Minimize the admission of containers with the NET_RAW capability (Automated)
**Rationale**
Those controls are defined to evaluate certain PodSecurityPolicy resources in the benchmark, but the PodSecurityPolicy API is removed starting in Kubernetes v1.25, therefore those controls fail in the scan.
**Remediation**
Pod Security Admission is configured by setting the Pod Security Admission Configuration Template (PSACT) `rancher-restricted` in the cluster to enforce the restriction on applicable namespaces.
Due to the lack of utilizing PSA in the current version of the benchmark, the above controls will not be addressed until an updated version of the benchmark is available.
</TabItem>
</Tabs>
## Reference Hardened RKE2 Template Configuration
The reference template configuration is used in Rancher to create a hardened RKE2 custom cluster. This reference does not include other required **cluster configuration** directives which will vary depending on your environment.
<Tabs groupId="rke2-version">
<TabItem value='v1.25 and Newer' default>
```yaml
apiVersion: provisioning.cattle.io/v1
kind: Cluster
metadata:
name: rke2-125-new
namespace: fleet-default
spec:
defaultPodSecurityAdmissionConfigurationTemplateName: rancher-restricted
kubernetesVersion: v1.25.6+rke2r1
rkeConfig:
etcd:
disableSnapshots: false
snapshotRetention: 5
snapshotScheduleCron: 0 */5 * * *
machineGlobalConfig:
cni: calico
disable-kube-proxy: false
etcd-expose-metrics: false
machineSelectorConfig:
- config:
profile: cis-1.23
protect-kernel-defaults: true
upgradeStrategy:
controlPlaneConcurrency: '1'
controlPlaneDrainOptions:
deleteEmptyDirData: true
disableEviction: false
enabled: false
force: false
gracePeriod: -1
ignoreDaemonSets: true
skipWaitForDeleteTimeoutSeconds: 0
timeout: 120
workerConcurrency: '1'
workerDrainOptions:
deleteEmptyDirData: true
disableEviction: false
enabled: false
force: false
gracePeriod: -1
ignoreDaemonSets: true
skipWaitForDeleteTimeoutSeconds: 0
timeout: 120
```
</TabItem>
<TabItem value='v1.22 up to 1.24'>
```yaml
apiVersion: provisioning.cattle.io/v1
kind: Cluster
metadata:
name: rke2-v124
namespace: fleet-default
spec:
defaultPodSecurityPolicyTemplateName: restricted-noroot
kubernetesVersion: v1.24.10+rke2r1
rkeConfig:
etcd:
disableSnapshots: false
snapshotRetention: 5
snapshotScheduleCron: 0 */5 * * *
machineGlobalConfig:
cni: calico
disable-kube-proxy: false
etcd-expose-metrics: false
machineSelectorConfig:
- config:
profile: cis-1.6
protect-kernel-defaults: true
upgradeStrategy:
controlPlaneConcurrency: '1'
controlPlaneDrainOptions:
deleteEmptyDirData: true
disableEviction: false
enabled: false
force: false
gracePeriod: -1
ignoreDaemonSets: true
skipWaitForDeleteTimeoutSeconds: 0
timeout: 120
workerConcurrency: '1'
workerDrainOptions:
deleteEmptyDirData: true
disableEviction: false
enabled: false
force: false
gracePeriod: -1
ignoreDaemonSets: true
skipWaitForDeleteTimeoutSeconds: 0
timeout: 120
```
</TabItem>
</Tabs>
## Conclusion
If you have followed this guide, your RKE2 custom cluster provisioned by Rancher will be configured to pass the CIS Kubernetes Benchmark. You can review our RKE2 self-assessment guides to understand how we verified each of the benchmarks and how you can do the same on your cluster.
docs/reference-guides/rancher-security/hardening-guides/rke2-self-assessment-guide-with-cis-v1.23-k8s-v1.23.md
+3196
View File
File diff suppressed because one or more lines are too long
docs/reference-guides/rancher-security/hardening-guides/rke2-self-assessment-guide-with-cis-v1.23-k8s-v1.24.md
+3196
View File
File diff suppressed because one or more lines are too long
docs/reference-guides/rancher-security/hardening-guides/rke2-self-assessment-guide-with-cis-v1.23-k8s-v1.25.md
+3196
View File
File diff suppressed because one or more lines are too long
docs/reference-guides/rancher-security/rancher-v2.6-hardening-guides/rke1-self-assessment-guide-with-cis-v1.6-benchmark.md
-3101
View File
File diff suppressed because it is too large Load Diff
docs/reference-guides/rancher-security/rancher-v2.6-hardening-guides/rke2-hardening-guide-with-cis-v1.6-benchmark.md
-409
View File
@@ -1,409 +0,0 @@
---
title: RKE2 Hardening Guide with CIS v1.6 Benchmark
---
This document provides prescriptive guidance for hardening a production installation of a RKE2 cluster to be provisioned with Rancher v2.6.5. It outlines the configurations and controls required to address Kubernetes benchmark controls from the Center for Information Security (CIS).
:::note
This hardening guide describes how to secure the nodes in your cluster, and it is recommended to follow this guide before installing Kubernetes.
:::
This hardening guide is intended to be used for RKE2 clusters and associated with specific versions of the CIS Kubernetes Benchmark, Kubernetes, and Rancher:
| Rancher Version | CIS Benchmark Version | Kubernetes Version |
| --------------- | --------------------- | ------------------ |
| Rancher v2.6.5+ | Benchmark v1.6 | Kubernetes v1.21 up to v1.23 |
[Click here to download a PDF version of this document](https://releases.rancher.com/documents/security/2.6/Rancher_RKE2_v2-6_CIS_v1-6_Hardening_Guide.pdf).
### Overview
This document provides prescriptive guidance for hardening a RKE2 cluster to be provisioned through Rancher v2.6.5+ with Kubernetes v1.21 up to v1.23. It outlines the configurations required to address Kubernetes benchmark controls from the Center for Information Security (CIS).
For more details about evaluating a hardened RKE2 cluster against the official CIS benchmark, refer to the [RKE2 - CIS 1.6 Benchmark - Self-Assessment Guide - Rancher v2.6](rke2-self-assessment-guide-with-cis-v1.6-benchmark.md).
RKE2 is designed to be "hardened by default" and pass the majority of the Kubernetes CIS controls without modification. There are a few notable exceptions to this that require manual intervention to fully pass the CIS Benchmark:
1. RKE2 will not modify the host operating system. Therefore, you, the operator, must make a few host-level modifications.
2. Certain CIS policy controls for `PodSecurityPolicies` and `NetworkPolicies` will restrict the functionality of the cluster. You must opt into having RKE2 configuring these out of the box.
To help ensure these above requirements are met, RKE2 can be started with the `profile` flag set to `cis-1.6`. This flag generally does two things:
1. Checks that host-level requirements have been met. If they haven't, RKE2 will exit with a fatal error describing the unmet requirements.
2. Configures runtime pod security policies and network policies that allow the cluster to pass associated controls.
:::note
The profile's flag only valid values are `cis-1.5` or `cis-1.6`. It accepts a string value to allow for other profiles in the future.
:::
The following section outlines the specific actions that are taken when the `profile` flag is set to `cis-1.6`.
### Host-level requirements
There are two areas of host-level requirements: kernel parameters and etcd process/directory configuration. These are outlined in this section.
#### Ensure `protect-kernel-defaults` is set
This is a kubelet flag that will cause the kubelet to exit if the required kernel parameters are unset or are set to values that are different from the kubelet's defaults.
When the `profile` flag is set, RKE2 will set the flag to `true`.
:::caution
`protect-kernel-defaults` is exposed as a configuration flag for RKE2. If you have set `profile` to "cis-1.x" and `protect-kernel-defaults` to `false` explicitly, RKE2 will exit with an error.
:::
RKE2 will also check the same kernel parameters that the kubelet does and exit with an error following the same rules as the kubelet. This is done as a convenience to help the operator more quickly and easily identify what kernel parameters are violating the kubelet defaults.
Both `protect-kernel-defaults` and `profile` flags can be set in RKE2 template configuration file.
```yaml
spec:
rkeConfig:
machineSelectorConfig:
- config:
profile: cis-1.6
protect-kernel-defaults: true
```
#### Ensure etcd is configured properly
The CIS Benchmark requires that the etcd data directory be owned by the `etcd` user and group. This implicitly requires the etcd process to be ran as the host-level `etcd` user. To achieve this, RKE2 takes several steps when started with a valid "cis-1.x" profile:
1. Check that the `etcd` user and group exists on the host. If they don't, exit with an error.
2. Create etcd's data directory with `etcd` as the user and group owner.
3. Ensure the etcd process is ran as the `etcd` user and group by setting the etcd static pod's `SecurityContext` appropriately.
### Setting up hosts
This section gives you the commands necessary to configure your host to meet the above requirements.
#### Set kernel parameters
The following `sysctl` configuration is recommended for all nodes type in the cluster. Set the following parameters in `/etc/sysctl.d/90-kubelet.conf`:
```ini
vm.panic_on_oom=0
vm.overcommit_memory=1
kernel.panic=10
kernel.panic_on_oops=1
```
Run `sudo sysctl -p /etc/sysctl.d/90-kubelet.conf` to enable the settings.
Please perform this step only on fresh installations, before actually deploying RKE2 through Rancher.
#### Create the etcd user
On some Linux distributions, the `useradd` command will not create a group. The `-U` flag is included below to account for that. This flag tells `useradd` to create a group with the same name as the user.
```bash
sudo useradd -r -c "etcd user" -s /sbin/nologin -M etcd -U
```
### Kubernetes runtime requirements
The runtime requirements to pass the CIS Benchmark are centered around pod security and network policies. These are outlined in this section.
#### `PodSecurityPolicies`
RKE2 always runs with the `PodSecurityPolicy` admission controller turned on. However, when it is **not** started with a valid "cis-1.x" profile, RKE2 will put an unrestricted policy in place that allows Kubernetes to run as though the `PodSecurityPolicy` admission controller was not enabled.
When ran with a valid "cis-1.x" profile, RKE2 will put a much more restrictive set of policies in place. These policies meet the requirements outlined in section 5.2 of the CIS Benchmark.
> The Kubernetes control plane components and critical additions such as CNI, DNS, and Ingress are ran as pods in the `kube-system` namespace. Therefore, this namespace will have a policy that is less restrictive so that these components can run properly.
#### `NetworkPolicies`
When ran with a valid "cis-1.x" profile, RKE2 will put `NetworkPolicies` in place that passes the CIS Benchmark for Kubernetes' built-in namespaces. These namespaces are: `kube-system`, `kube-public`, `kube-node-lease`, and `default`.
The `NetworkPolicy` used will only allow pods within the same namespace to talk to each other. The notable exception to this is that it allows DNS requests to be resolved.
:::note
Operators must manage network policies as normal for additional namespaces that are created.
:::
#### Configure `default` service account
**Set `automountServiceAccountToken` to `false` for `default` service accounts**
Kubernetes provides a `default` service account which is used by cluster workloads where no specific service account is assigned to the pod. Where access to the Kubernetes API from a pod is required, a specific service account should be created for that pod, and rights granted to that service account. The `default` service account should be configured such that it does not provide a service account token and does not have any explicit rights assignments.
For each namespace including `default` and `kube-system` on a standard RKE2 install, the `default` service account must include this value:
```yaml
automountServiceAccountToken: false
```
For namespaces created by the cluster operator, the following script and configuration file can be used to configure the `default` service account.
The configuration bellow must be saved to a file called `account_update.yaml`.
```yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: default
automountServiceAccountToken: false
```
Create a bash script file called `account_update.sh`. Be sure to `sudo chmod +x account_update.sh` so the script has execute permissions.
```bash
#!/bin/bash -e
for namespace in $(kubectl get namespaces -A -o=jsonpath="{.items[*]['metadata.name']}"); do
echo -n "Patching namespace $namespace - "
kubectl patch serviceaccount default -n ${namespace} -p "$(cat account_update.yaml)"
done
```
Execute this script to apply the `account_update.yaml` configuration to `default` service account in all namespaces.
### API Server audit configuration
CIS requirements 1.2.22 to 1.2.25 are related to configuring audit logs for the API Server. When RKE2 is started with the `profile` flag set to `cis-1.6`, it will automatically configure hardened `--audit-log-` parameters in the API Server to pass those CIS checks.
RKE2's default audit policy is configured to not log requests in the API Server. This is done to allow cluster operators flexibility to customize an audit policy that suits their auditing requirements and needs, as these are specific to each users' environment and policies.
A default audit policy is created by RKE2 when started with the `profile` flag set to `cis-1.6`. The policy is defined in `/etc/rancher/rke2/audit-policy.yaml`.
```yaml
apiVersion: audit.k8s.io/v1
kind: Policy
metadata:
creationTimestamp: null
rules:
- level: None
```
To start logging requests to the API Server, at least `level` parameter must be modified, for example, to `Metadata`. Detailed information about policy configuration for the API server can be found in the Kubernetes [documentation](https://kubernetes.io/docs/tasks/debug-application-cluster/audit/).
After adapting the audit policy, RKE2 must be restarted to load the new configuration.
```shell
sudo systemctl restart rke2-server.service
```
API Server audit logs will be written to `/var/lib/rancher/rke2/server/logs/audit.log`.
### Known issues
The following are controls that RKE2 currently does not pass. Each gap will be explained and whether it can be passed through manual operator intervention or if it will be addressed in a future release.
#### Control 1.1.12
Ensure that the etcd data directory ownership is set to `etcd:etcd`.
**Rationale**
etcd is a highly-available key-value store used by Kubernetes deployments for persistent storage of all of its REST API objects. This data directory should be protected from any unauthorized reads or writes. It should be owned by `etcd:etcd`.
**Remediation**
This can be remediated by creating an `etcd` user and group as described above.
#### Control 5.1.5
Ensure that default service accounts are not actively used
**Rationale** Kubernetes provides a `default` service account which is used by cluster workloads where no specific service account is assigned to the pod.
Where access to the Kubernetes API from a pod is required, a specific service account should be created for that pod, and rights granted to that service account.
The `default` service account should be configured such that it does not provide a service account token and does not have any explicit rights assignments.
This can be remediated by updating the `automountServiceAccountToken` field to `false` for the `default` service account in each namespace.
**Remediation**
You can manually update this field on service accounts in your cluster to pass the control as described above.
#### Control 5.3.2
Ensure that all Namespaces have Network Policies defined
**Rationale**
Running different applications on the same Kubernetes cluster creates a risk of one compromised application attacking a neighboring application. Network segmentation is important to ensure that containers can communicate only with those they are supposed to. A network policy is a specification of how selections of pods are allowed to communicate with each other and other network endpoints.
Network Policies are namespace scoped. When a network policy is introduced to a given namespace, all traffic not allowed by the policy is denied. However, if there are no network policies in a namespace all traffic will be allowed into and out of the pods in that namespace.
**Remediation**
This can be remediated by setting `profile: "cis-1.6"` in RKE2 template configuration file. An example can be found below.
### Reference Hardened RKE2 Template Configuration
The reference template configuration is used in Rancher to create a hardened RKE2 custom cluster. This reference does not include other required **cluster configuration** directives which will vary depending on your environment.
```yaml
apiVersion: provisioning.cattle.io/v1
kind: Cluster
metadata:
name: <replace_with_cluster_name>
annotations:
{}
# key: string
labels:
{}
# key: string
namespace: fleet-default
spec:
defaultPodSecurityPolicyTemplateName: ''
kubernetesVersion: <replace_with_kubernetes_version>
localClusterAuthEndpoint:
caCerts: ''
enabled: false
fqdn: ''
rkeConfig:
chartValues:
rke2-canal:
{}
etcd:
disableSnapshots: false
s3:
# bucket: string
# cloudCredentialName: string
# endpoint: string
# endpointCA: string
# folder: string
# region: string
# skipSSLVerify: boolean
snapshotRetention: 5
snapshotScheduleCron: 0 */5 * * *
machineGlobalConfig:
cni: canal
machinePools:
# - cloudCredentialSecretName: string
# controlPlaneRole: boolean
# displayName: string
# drainBeforeDelete: boolean
# etcdRole: boolean
# labels:
# key: string
# machineConfigRef:
# apiVersion: string
# fieldPath: string
# kind: string
# name: string
# namespace: string
# resourceVersion: string
# uid: string
# machineDeploymentAnnotations:
# key: string
# machineDeploymentLabels:
# key: string
# machineOS: string
# maxUnhealthy: string
# name: string
# nodeStartupTimeout: string
# paused: boolean
# quantity: int
# rollingUpdate:
# maxSurge: string
# maxUnavailable: string
# taints:
# - effect: string
# key: string
# timeAdded: string
# value: string
# unhealthyNodeTimeout: string
# unhealthyRange: string
# workerRole: boolean
machineSelectorConfig:
- config:
profile: cis-1.6
protect-kernel-defaults: true
# - config:
#
# machineLabelSelector:
# matchExpressions:
# - key: string
# operator: string
# values:
# - string
# matchLabels:
# key: string
registries:
configs:
{}
#authConfigSecretName: string
# caBundle: string
# insecureSkipVerify: boolean
# tlsSecretName: string
mirrors:
{}
#endpoint:
# - string
# rewrite:
# key: string
upgradeStrategy:
controlPlaneConcurrency: 10%
controlPlaneDrainOptions:
# deleteEmptyDirData: boolean
# disableEviction: boolean
# enabled: boolean
# force: boolean
# gracePeriod: int
# ignoreDaemonSets: boolean
# ignoreErrors: boolean
# postDrainHooks:
# - annotation: string
# preDrainHooks:
# - annotation: string
# skipWaitForDeleteTimeoutSeconds: int
# timeout: int
workerConcurrency: 10%
workerDrainOptions:
# deleteEmptyDirData: boolean
# disableEviction: boolean
# enabled: boolean
# force: boolean
# gracePeriod: int
# ignoreDaemonSets: boolean
# ignoreErrors: boolean
# postDrainHooks:
# - annotation: string
# preDrainHooks:
# - annotation: string
# skipWaitForDeleteTimeoutSeconds: int
# timeout: int
# additionalManifest: string
# etcdSnapshotCreate:
# generation: int
# etcdSnapshotRestore:
# generation: int
# name: string
# restoreRKEConfig: string
# infrastructureRef:
# apiVersion: string
# fieldPath: string
# kind: string
# name: string
# namespace: string
# resourceVersion: string
# uid: string
# provisionGeneration: int
# rotateCertificates:
# generation: int
# services:
# - string
# rotateEncryptionKeys:
# generation: int
machineSelectorConfig:
- config: {}
# agentEnvVars:
# - name: string
# value: string
# cloudCredentialSecretName: string
# clusterAPIConfig:
# clusterName: string
# defaultClusterRoleForProjectMembers: string
# enableNetworkPolicy: boolean
# redeploySystemAgentGeneration: int
__clone: true
```
### Conclusion
If you have followed this guide, your RKE2 custom cluster provisioned by Rancher will be configured to pass the CIS Kubernetes Benchmark. You can review our RKE2 CIS Benchmark Self-Assessment Guide [v1.6](rke2-self-assessment-guide-with-cis-v1.6-benchmark.md) to understand how we verified each of the benchmarks and how you can do the same on your cluster.
docs/reference-guides/rancher-security/rancher-v2.6-hardening-guides/rke2-self-assessment-guide-with-cis-v1.6-benchmark.md
-3329
View File
File diff suppressed because it is too large Load Diff
docs/reference-guides/rancher-security/rancher-v2.7-hardening-guides/k3s-hardening-guide-with-cis-benchmark.md
-549
View File
@@ -1,549 +0,0 @@
---
title: K3s Hardening Guide with CIS Benchmark
---
This document provides prescriptive guidance for hardening a production installation of a K3s cluster to be provisioned with Rancher v2.7. It outlines the configurations and controls required to address Kubernetes benchmark controls from the Center for Information Security (CIS).
:::note
This hardening guide describes how to secure the nodes in your cluster, and it is recommended to follow this guide before installing Kubernetes.
:::
This hardening guide is intended to be used for K3s clusters and associated with specific versions of the CIS Kubernetes Benchmark, Kubernetes, and Rancher:
| Rancher Version | CIS Benchmark Version | Kubernetes Version |
| --------------- | --------------------- | ------------------ |
| Rancher v2.7 | Benchmark v1.20 | Kubernetes v1.21 |
| Rancher v2.7 | Benchmark v1.23 | Kubernetes v1.22 up to v1.24 |
### Overview
For more details about evaluating a hardened K3s cluster against the official CIS benchmark, refer to K3s - CIS Benchmark - Self-Assessment Guide - Rancher v2.7 for [CIS v1.20](k3s-self-assessment-guide-with-cis-v1.20-benchmark.md) and [CIS v1.23](k3s-self-assessment-guide-with-cis-v1.23-benchmark.md).
K3s has a number of security mitigations applied and turned on by default and will pass a number of the Kubernetes CIS controls without modification. There are some notable exceptions to this that require manual intervention to fully comply with the CIS Benchmark:
1. K3s will not modify the host operating system. Any host-level modifications will need to be done manually.
2. Certain CIS policy controls for `PodSecurityPolicies` and `NetworkPolicies` will restrict the functionality of the cluster. You must opt into having K3s configure these by adding the appropriate options (enabling of admission plugins) to your command-line flags or configuration file as well as manually applying appropriate policies. Further details are presented in the sections below.
The first section (1.1) of the CIS Benchmark concerns itself primarily with pod manifest permissions and ownership. K3s doesn't utilize these for the core components since everything is packaged into a single binary.
## Host-level Requirements
### Ensure `protect-kernel-defaults` is set (control 4.2.6)
This is a kubelet flag that will cause the kubelet to exit if the required kernel parameters are unset or are set to values that are different from the kubelet's defaults.
This can be remediated by adding the following argument line to K3s cluster configuration file:
```yaml
spec:
rkeConfig:
machineSelectorConfig:
- config:
protect-kernel-defaults: true # Control 4.2.6
```
#### Set kernel parameters
Create a file called `/etc/sysctl.d/90-kubelet.conf` and add the snippet below. Then run `sysctl -p /etc/sysctl.d/90-kubelet.conf`.
```bash
vm.panic_on_oom=0
vm.overcommit_memory=1
kernel.panic=10
kernel.panic_on_oops=1
```
This configuration needs to be done before setting the kubelet flag, otherwise K3s will fail to start.
## Kubernetes Runtime Requirements
The runtime requirements to comply with the CIS Benchmark are centered around pod security policies (PSPs) and its admission control plugin, network policies and API Server auditing logs. These are outlined in this section. K3s doesn't apply any default PSPs or network policies. However, K3s ships with a controller that is meant to apply a given set of network policies. By default, K3s runs with the `NodeRestriction` admission controller. To enable PSPs, add the following line to K3s cluster configuration file:
```yaml
spec:
rkeConfig:
machineGlobalConfig:
kube-apiserver-arg:
- enable-admission-plugins=NodeRestriction,PodSecurityPolicy # CIS 1.2.16 and CIS 5.2
```
This will have the effect of maintaining the `NodeRestriction` plugin as well as enabling the `PodSecurityPolicy`.
### Pod Security Policies (control 5.2)
When PSPs are enabled, a policy can be applied to satisfy the necessary controls described in section 5.2 of the CIS Benchmark.
Here is an example of a compliant PSP.
```yaml
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
name: restricted-psp
spec:
privileged: false # CIS - 5.2.1
allowPrivilegeEscalation: false # CIS - 5.2.5
requiredDropCapabilities: # CIS - 5.2.7/8/9
- ALL
volumes:
- 'configMap'
- 'emptyDir'
- 'projected'
- 'secret'
- 'downwardAPI'
- 'csi'
- 'persistentVolumeClaim'
- 'ephemeral'
hostNetwork: false # CIS - 5.2.4
hostIPC: false # CIS - 5.2.3
hostPID: false # CIS - 5.2.2
runAsUser:
rule: 'MustRunAsNonRoot' # CIS - 5.2.6
seLinux:
rule: 'RunAsAny'
supplementalGroups:
rule: 'MustRunAs'
ranges:
- min: 1
max: 65535
fsGroup:
rule: 'MustRunAs'
ranges:
- min: 1
max: 65535
readOnlyRootFilesystem: false
```
For the above PSP to be effective, we need to create a `ClusterRole` and a `ClusterRoleBinding`. We also need to include a "system unrestricted policy" which is needed for system-level pods that require additional privileges, as exemplified below.
```yaml
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
name: restricted-psp
spec:
privileged: false
allowPrivilegeEscalation: false
requiredDropCapabilities:
- ALL
volumes:
- 'configMap'
- 'emptyDir'
- 'projected'
- 'secret'
- 'downwardAPI'
- 'csi'
- 'persistentVolumeClaim'
- 'ephemeral'
hostNetwork: false
hostIPC: false
hostPID: false
runAsUser:
rule: 'MustRunAsNonRoot'
seLinux:
rule: 'RunAsAny'
supplementalGroups:
rule: 'MustRunAs'
ranges:
- min: 1
max: 65535
fsGroup:
rule: 'MustRunAs'
ranges:
- min: 1
max: 65535
readOnlyRootFilesystem: false
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: psp:restricted-psp
labels:
addonmanager.kubernetes.io/mode: EnsureExists
rules:
- apiGroups: ['extensions']
resources: ['podsecuritypolicies']
verbs: ['use']
resourceNames:
- restricted-psp
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: default:restricted-psp
labels:
addonmanager.kubernetes.io/mode: EnsureExists
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: psp:restricted-psp
subjects:
- kind: Group
name: system:authenticated
apiGroup: rbac.authorization.k8s.io
---
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
name: system-unrestricted-psp
spec:
allowPrivilegeEscalation: true
allowedCapabilities:
- '*'
fsGroup:
rule: RunAsAny
hostIPC: true
hostNetwork: true
hostPID: true
hostPorts:
- max: 65535
min: 0
privileged: true
runAsUser:
rule: RunAsAny
seLinux:
rule: RunAsAny
supplementalGroups:
rule: RunAsAny
volumes:
- '*'
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: system-unrestricted-node-psp-rolebinding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system-unrestricted-psp-role
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: Group
name: system:nodes
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: system-unrestricted-psp-role
rules:
- apiGroups:
- policy
resourceNames:
- system-unrestricted-psp
resources:
- podsecuritypolicies
verbs:
- use
```
The policy file `policy.yaml` can be placed in the `/var/lib/rancher/k3s/server/manifests` directory. Both the policy file and the its directory hierarchy, ideally, must be created before starting K3s. A restrictive access permission is recommended to avoid leaking potential sensitive information.
```bash
sudo mkdir -p -m 700 /var/lib/rancher/k3s/server/manifests
```
### Network Policies (control 5.3.2)
CIS requires that all namespaces have a network policy applied that reasonably limits traffic into namespaces and pods.
:::note
This is a manual check in the CIS Benchmark. The CIS scan will flag the result as `warning`, because manual inspection is necessary by the cluster operator.
:::
Here is an example of a compliant network policy.
```yaml
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
name: intra-namespace
namespace: kube-system
spec:
podSelector: {}
ingress:
- from:
- namespaceSelector:
matchLabels:
name: kube-system
```
:::note
Kubernetes' additions such as CNI, DNS, and Ingress are ran as pods in the `kube-system` namespace. Therefore, this namespace will have a policy that is less restrictive so that these components can run properly.
:::
With the applied restrictions, DNS will be blocked unless purposely allowed. Below is a network policy that will allow DNS related traffic.
```yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-network-dns-policy
namespace: <NAMESPACE>
spec:
ingress:
- ports:
- port: 53
protocol: TCP
- port: 53
protocol: UDP
podSelector:
matchLabels:
k8s-app: kube-dns
policyTypes:
- Ingress
```
The metrics-server and Traefik ingress controller will be blocked by default if network policies are not created to allow access.
```yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-all-metrics-server
namespace: kube-system
spec:
podSelector:
matchLabels:
k8s-app: metrics-server
ingress:
- {}
policyTypes:
- Ingress
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-all-svclbtraefik-ingress
namespace: kube-system
spec:
podSelector:
matchLabels:
svccontroller.k3s.cattle.io/svcname: traefik
ingress:
- {}
policyTypes:
- Ingress
```
:::note
Operators must manage network policies as normal for additional namespaces that are created.
:::
The network policies can be added in the same policy file used for PSPs in `/var/lib/rancher/k3s/server/manifests/policy.yaml` or on its own file.
### API Server audit configuration
CIS requirements v1.20 - 1.2.22 to 1.2.25 and v1.23 - 1.2.19 to 1.2.22 are related to configuring audit logs for the API Server. K3s doesn't create by default the log directory and audit policy, as auditing requirements are specific to each user's policies and environment.
The log directory, ideally, must be created before starting K3s. A restrictive access permission is recommended to avoid leaking potential sensitive information.
```bash
sudo mkdir -p -m 700 /var/lib/rancher/k3s/server/logs
```
A starter audit policy to log request metadata is provided below. The policy should be written to a file named `audit.yaml` in `/var/lib/rancher/k3s/server` directory. Detailed information about policy configuration for the API server can be found in the Kubernetes [documentation](https://kubernetes.io/docs/tasks/debug/debug-cluster/audit/).
```yaml
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
- level: Metadata
```
Further configurations, as described below, are also needed to pass CIS checks and are not configured by default in K3s, because they vary depending on the users' environment and needs:
- Ensure that the `--audit-log-path` argument is set.
- Ensure that the `--audit-log-maxage` argument is set to 30 or as appropriate.
- Ensure that the `--audit-log-maxbackup` argument is set to 10 or as appropriate.
- Ensure that the `--audit-log-maxsize` argument is set to 100 or as appropriate.
To enable and configure audit logs, add the following line to K3s cluster configuration file:
```yaml
spec:
rkeConfig:
machineGlobalConfig:
kube-apiserver-arg:
- audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml # CIS v1.20/v1.23 3.2.1
- audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log # CIS v1.20 1.2.21 and CIS v1.23 1.2.19
- audit-log-maxage=30 # CIS v1.20 1.2.22 and CIS v1.23 1.2.20
- audit-log-maxbackup=10 # CIS v1.20 1.2.23 and CIS v1.23 1.2.21
- audit-log-maxsize=100 # CIS v1.20 1.2.24 and CIS v1.23 1.2.22
```
## Known Issues
The following are controls that K3s currently does not pass by default. Each gap will be explained, along with a note clarifying whether it can be passed through manual operator intervention, or if it will be addressed in a future release of K3s.
### Control CIS v1.20 - 1.2.13 / CIS v1.23 - 1.2.14
Ensure that the admission control plugin `ServiceAccount` is set
<details>
<summary>Rationale</summary>
Follow the documentation and create `ServiceAccount` objects as per your environment. Then, edit the API server pod specification file $apiserverconf
on the control plane node and ensure that the `--disable-admission-plugins` parameter is set to a value that does not include `ServiceAccount`.
This can be remediated by adding the following argument line to K3s cluster configuration file:
```yaml
spec:
rkeConfig:
machineGlobalConfig:
kube-apiserver-arg:
kube-apiserver-arg:
- enable-admission-plugins=ServiceAccount # CIS v1.20 - 1.2.13 / CIS v1.23 - 1.2.14
```
</details>
### Control CIS v1.20 1.2.26 and CIS v1.23 1.2.23
Ensure that the `--request-timeout` argument is set as appropriate.
<details>
<summary>Rationale</summary>
Setting global request timeout allows extending the API server request timeout limit to a duration appropriate to the user's connection speed. By default, it is set to 60 seconds which might be problematic on slower connections making cluster resources inaccessible once the data volume for requests exceeds what can be transmitted in 60 seconds. But, setting this timeout limit to be too large can exhaust the API server resources making it prone to Denial-of-Service attack. Hence, it is recommended to set this limit as appropriate and change the default limit of 60 seconds only if needed.
This can be remediated by adding the following argument line to K3s cluster configuration file:
```yaml
spec:
rkeConfig:
machineGlobalConfig:
kube-apiserver-arg:
- request-timeout=300s # Control CIS v1.20 1.2.26 and CIS v1.23 1.2.23
```
</details>
### Control CIS v1.23 1.2.24
Ensure that the `--service-account-lookup` argument is set to true.
<details>
<summary>Rationale</summary>
If `--service-account-lookup` is not enabled, the apiserver only verifies that the authentication token is valid, and does not validate that the service account token mentioned in the request is actually present in etcd. This allows using a service account token even after the corresponding service account is deleted. This is an example of time of check to time of use security issue.
This can be remediated by adding the following argument line to K3s cluster configuration file:
```yaml
spec:
rkeConfig:
machineGlobalConfig:
kube-apiserver-arg:
- service-account-lookup=true # Control CIS v1.23 1.2.24
```
</details>
### CIS v1.20 1.2.32 and CIS v1.23 1.2.30
Ensure that the `--encryption-provider-config` argument is set as appropriate.
<details>
<summary>Rationale</summary>
`etcd` is a highly available key-value store used by Kubernetes deployments for persistent storage of all of its REST API objects. These objects are sensitive in nature and should be encrypted at rest to avoid any disclosures.
Detailed steps on how to configure secrets encryption in K3s are available in [K3s Secrets Encryption Config](https://docs.k3s.io/security/secrets-encryption).
</details>
### Control CIS v1.20 1.2.33 and CIS v1.23 1.2.31
Ensure that encryption providers are appropriately configured.
<details>
<summary>Rationale</summary>
Where `etcd` encryption is used, it is important to ensure that the appropriate set of encryption providers is used. Currently, the `aescbc`, `kms` and `secretbox` are likely to be appropriate options.
This can be remediated by passing a valid configuration to `k3s` as outlined above. Detailed steps on how to configure secrets encryption in K3s are available in [K3s Secrets Encryption Config](https://docs.k3s.io/security/secrets-encryption).
</details>
### Control 4.2.7
Ensure that the `--make-iptables-util-chains` argument is set to true.
<details>
<summary>Rationale</summary>
Kubelets can automatically manage the required changes to iptables based on how you choose your networking options for the pods. It is recommended to let kubelets manage the changes to iptables. This ensures that the iptables configuration remains in sync with pods networking configuration. Manually configuring iptables with dynamic pod network configuration changes might hamper the communication between pods/containers and to the outside world. You might have iptables rules too restrictive or too open.
This can be remediated by adding the following argument line to K3s cluster configuration file:
```yaml
spec:
rkeConfig:
machineSelectorConfig:
- config:
kubelet-arg:
- make-iptables-util-chains=true # Control 4.2.7
```
</details>
### Control 5.1.5
Ensure that default service accounts are not actively used
<details>
<summary>Rationale</summary>
Kubernetes provides a `default` service account which is used by cluster workloads where no specific service account is assigned to the pod.
Where access to the Kubernetes API from a pod is required, a specific service account should be created for that pod, and rights granted to that service account.
The default service account should be configured such that it does not provide a service account token and does not have any explicit rights assignments.
This can be remediated by updating the `automountServiceAccountToken` field to `false` for the `default` service account in each namespace.
For `default` service accounts in the built-in namespaces (`kube-system`, `kube-public`, `kube-node-lease`, and `default`), K3s does not automatically do this. You can manually update this field on these service accounts to pass the control or use the script below to automate this task.
Save the follow configuration to a file called `account_update.yaml`.
```yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: default
automountServiceAccountToken: false
```
Create a bash script file called `account_update.sh`. Be sure to `sudo chmod +x account_update.sh` so the script has execute permissions.
```bash
#!/bin/bash -e
for namespace in $(kubectl get namespaces -A -o=jsonpath="{.items[*]['metadata.name']}"); do
echo -n "Patching namespace $namespace - "
kubectl patch serviceaccount default -n ${namespace} -p "$(cat account_update.yaml)"
done
```
Execute the script to update the default service account in each namespace.
</details>
### Reference Hardened K3s Template Configuration
The reference template configuration below is used in Rancher to create a hardened K3s custom cluster based on each CIS control presented in this guide. This reference does not include other required **cluster configuration** directives which will vary depending on your environment.
```yaml
apiVersion: provisioning.cattle.io/v1
kind: Cluster
metadata:
name: # Define cluster name
annotations:
spec:
defaultPodSecurityPolicyTemplateName: '' # Define the PSP policy to use
enableNetworkPolicy: true
kubernetesVersion: # Define K3s version
rkeConfig:
machineGlobalConfig:
kube-apiserver-arg:
- enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount # CIS 1.2.16, CIS 5.2 and CIS v1.20 - 1.2.13 / CIS v1.23 - 1.2.14
- audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml # CIS v1.20/v1.23 3.2.1
- audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log # CIS v1.20 1.2.21 and CIS v1.23 1.2.19
- audit-log-maxage=30 # CIS v1.20 1.2.22 and CIS v1.23 1.2.20
- audit-log-maxbackup=10 # CIS v1.20 1.2.23 and CIS v1.23 1.2.21
- audit-log-maxsize=100 # CIS v1.20 1.2.24 and CIS v1.23 1.2.22
- request-timeout=300s # Control CIS v1.20 1.2.26 and CIS v1.23 1.2.23
- service-account-lookup=true # Control CIS v1.23 1.2.24
secrets-encryption: true
machineSelectorConfig:
- config:
kubelet-arg:
- make-iptables-util-chains=true # Control 4.2.7
protect-kernel-defaults: true # Control 4.2.6
```
### Conclusion
If you have followed this guide, your K3s custom cluster provisioned by Rancher will be configured to pass the CIS Kubernetes Benchmark. You can review our K3s CIS Benchmark Self-Assessment Guide for [CIS v1.20](k3s-self-assessment-guide-with-cis-v1.20-benchmark.md) and [CIS v1.23](k3s-self-assessment-guide-with-cis-v1.23-benchmark.md) to understand how we verified each of the benchmarks and how you can do the same on your cluster.
docs/reference-guides/rancher-security/rancher-v2.7-hardening-guides/k3s-self-assessment-guide-with-cis-v1.20-benchmark.md
-3133
View File
File diff suppressed because it is too large Load Diff
docs/reference-guides/rancher-security/rancher-v2.7-hardening-guides/k3s-self-assessment-guide-with-cis-v1.23-benchmark.md
-3143
View File
File diff suppressed because it is too large Load Diff
docs/security/hardening-guides/rke-1.6-benchmark-2.6/Rancher_v2-6_CIS_v1-6_Benchmark_Assessment.pdf
-14862
View File
File diff suppressed because one or more lines are too long
docs/security/hardening-guides/rke-1.6-hardening-2.6/Rancher_v2-6_CIS_v1-6_Hardening_Guide.pdf
-8259
View File
File diff suppressed because one or more lines are too long
docs/security/hardening-guides/rke2-1.6-benchmark-2.6/Rancher_RKE2_v2-6_CIS_v1-6_Benchmark_Assessment.pdf
-14240
View File
File diff suppressed because one or more lines are too long
docs/security/hardening-guides/rke2-1.6-hardening-2.6/Rancher_RKE2_v2-6_CIS_v1-6_Hardening_Guide.pdf
-4149
View File
File diff suppressed because one or more lines are too long
sidebars.js
+14 -19
View File
@@ -1049,29 +1049,24 @@ const sidebars = {
items: [
{
type: 'category',
label: 'Rancher v2.6 Hardening Guides',
label: 'Hardening Guides',
link: {
type: 'doc',
id: "pages-for-subheaders/rancher-v2.6-hardening-guides",
id: "pages-for-subheaders/rancher-hardening-guides",
},
items: [
"reference-guides/rancher-security/rancher-v2.6-hardening-guides/rke1-hardening-guide-with-cis-v1.6-benchmark",
"reference-guides/rancher-security/rancher-v2.6-hardening-guides/rke1-self-assessment-guide-with-cis-v1.6-benchmark",
"reference-guides/rancher-security/rancher-v2.6-hardening-guides/rke2-hardening-guide-with-cis-v1.6-benchmark",
"reference-guides/rancher-security/rancher-v2.6-hardening-guides/rke2-self-assessment-guide-with-cis-v1.6-benchmark",
],
},
{
type: 'category',
label: 'Rancher v2.7 Hardening Guides',
link: {
type: 'doc',
id: "pages-for-subheaders/rancher-v2.7-hardening-guides",
},
items: [
"reference-guides/rancher-security/rancher-v2.7-hardening-guides/k3s-hardening-guide-with-cis-benchmark",
"reference-guides/rancher-security/rancher-v2.7-hardening-guides/k3s-self-assessment-guide-with-cis-v1.20-benchmark",
"reference-guides/rancher-security/rancher-v2.7-hardening-guides/k3s-self-assessment-guide-with-cis-v1.23-benchmark",
"reference-guides/rancher-security/hardening-guides/rke1-hardening-guide",
"reference-guides/rancher-security/hardening-guides/rke1-self-assessment-guide-with-cis-v1.23-k8s-v1.23",
"reference-guides/rancher-security/hardening-guides/rke1-self-assessment-guide-with-cis-v1.23-k8s-v1.24",
"reference-guides/rancher-security/hardening-guides/rke1-self-assessment-guide-with-cis-v1.23-k8s-v1.25",
"reference-guides/rancher-security/hardening-guides/rke2-hardening-guide",
"reference-guides/rancher-security/hardening-guides/rke2-self-assessment-guide-with-cis-v1.23-k8s-v1.23",
"reference-guides/rancher-security/hardening-guides/rke2-self-assessment-guide-with-cis-v1.23-k8s-v1.24",
"reference-guides/rancher-security/hardening-guides/rke2-self-assessment-guide-with-cis-v1.23-k8s-v1.25",
"reference-guides/rancher-security/hardening-guides/k3s-hardening-guide",
"reference-guides/rancher-security/hardening-guides/k3s-self-assessment-guide-with-cis-v1.23-k8s-v1.23",
"reference-guides/rancher-security/hardening-guides/k3s-self-assessment-guide-with-cis-v1.23-k8s-v1.24",
"reference-guides/rancher-security/hardening-guides/k3s-self-assessment-guide-with-cis-v1.23-k8s-v1.25",
],
},
{