Updated hardening and benchmark guides to build RKE clusters CIS compliant

2026-09-25 04:28:15 +00:00 · 2019-12-17 21:51:28 +01:00
parent 5425c16f08
commit 0692d9a706
5 changed files with 3955 additions and 13 deletions
@@ -22,11 +22,12 @@ weight: 7505

 ### Rancher Hardening Guide

-The Rancher Hardening Guide is based off of controls and best practices found in the [CIS Kubernetes Benchmark](https://www.cisecurity.org/benchmark/kubernetes/) from the Center for Internet Security. The hardening guide provides prescriptive guidance for hardening a production installation of Rancher v2.1.x, v2.2.x and v.2.3.x. See Rancher's [Self Assessment of the CIS Kubernetes Benchmark](#cis-benchmark-rancher-self-assessment) for the full list of security controls.
+The Rancher Hardening Guide is based off of controls and best practices found in the [CIS Kubernetes Benchmark](https://www.cisecurity.org/benchmark/kubernetes/) from the Center for Internet Security. The hardening guide provides prescriptive guidance for hardening a production installation of Rancher v2.1.x, v2.2.x, v.2.3.0-v2.3.2 and v2.3.3+. See Rancher's [Self Assessment of the CIS Kubernetes Benchmark](#cis-benchmark-rancher-self-assessment) for the full list of security controls.

 - [Hardening Guide for Rancher v2.1.x with Kubernetes 1.11]({{< baseurl >}}/rancher/v2.x/en/security/hardening-2.1/)
 - [Hardening Guide for Rancher v2.2.x with Kubernetes 1.13]({{< baseurl >}}/rancher/v2.x/en/security/hardening-2.2/)
- [Hardening Guide for Rancher v2.3.x with Kubernetes 1.15]({{< baseurl >}}/rancher/v2.x/en/security/hardening-2.3/)
+- [Hardening Guide for Rancher v2.3.0-v2.3.2 with Kubernetes 1.15]({{< baseurl >}}/rancher/v2.x/en/security/hardening-2.3/)
+- [Hardening Guide for Rancher v2.3.3+ with Kubernetes 1.16]({{< baseurl >}}/rancher/v2.x/en/security/hardening-2.3.3/)

 ### CIS Benchmark Rancher Self-Assessment

@@ -34,10 +35,11 @@ The benchmark self-assessment is a companion to the Rancher security hardening g

 Because Rancher and RKE install Kubernetes services as Docker containers, many of the control verification checks in the CIS Kubernetes Benchmark don't apply. This guide will walk through the various controls and provide updated example commands to audit compliance in Rancher created clusters.  The original benchmark documents can be downloaded from the [CIS website](https://www.cisecurity.org/benchmark/kubernetes/).

- [CIS Kubernetes Benchmark 1.3.0 - Rancher 2.1.x with Kubernetes 1.11]({{< baseurl >}}/rancher/v2.x/en/security/benchmark-2.1/)
- [CIS Kubernetes Benchmark 1.4.0 - Rancher 2.2.x with Kubernetes 1.13]({{< baseurl >}}/rancher/v2.x/en/security/benchmark-2.2/#cis-kubernetes-benchmark-1-4-0-rancher-2-2-x-with-kubernetes-1-13/)
- [CIS Kubernetes Benchmark 1.4.1 - Rancher 2.2.x with Kubernetes 1.13]({{< baseurl >}}/rancher/v2.x/en/security/benchmark-2.2/#cis-kubernetes-benchmark-1-4-1-rancher-2-2-x-with-kubernetes-1-13)
- [CIS Kubernetes Benchmark 1.4.1 - Rancher 2.3.x with Kubernetes 1.15]({{< baseurl >}}/rancher/v2.x/en/security/benchmark-2.3/#cis-kubernetes-benchmark-1-4-1-rancher-2-3-x-with-kubernetes-1-15)
+* [CIS Kubernetes Benchmark 1.3.0 - Rancher 2.1.x with Kubernetes 1.11]({{< baseurl >}}/rancher/v2.x/en/security/benchmark-2.1/)
+* [CIS Kubernetes Benchmark 1.4.0 - Rancher 2.2.x with Kubernetes 1.13]({{< baseurl >}}/rancher/v2.x/en/security/benchmark-2.2/#cis-kubernetes-benchmark-1-4-0-rancher-2-2-x-with-kubernetes-1-13/)
+* [CIS Kubernetes Benchmark 1.4.1 - Rancher 2.2.x with Kubernetes 1.13]({{< baseurl >}}/rancher/v2.x/en/security/benchmark-2.2/#cis-kubernetes-benchmark-1-4-1-rancher-2-2-x-with-kubernetes-1-13)
+* [CIS Kubernetes Benchmark 1.4.1 - Rancher 2.3.0-2.3.2 with Kubernetes 1.15]({{< baseurl >}}/rancher/v2.x/en/security/benchmark-2.3/#cis-kubernetes-benchmark-1-4-1-rancher-2-3-0-2-3-2-with-kubernetes-1-15)
+* [CIS Kubernetes Benchmark 1.4.1 - Rancher 2.3.3+ with Kubernetes 1.16]({{< baseurl >}}/rancher/v2.x/en/security/benchmark-2.3.3/#cis-kubernetes-benchmark-1-4-1-rancher-2-3-3-+-with-kubernetes-1-16)

 ### Third Party Pen Test Reports

@@ -1,9 +1,9 @@
 ---
-title: CIS Benchmark Rancher Self-Assessment Guide - Rancher v2.3.x
+title: CIS Benchmark Rancher Self-Assessment Guide - Rancher v2.3.0-v2.3.2
 weight: 103
 ---

-### CIS Kubernetes Benchmark 1.4.1 - Rancher 2.3.x with Kubernetes 1.15
+### CIS Kubernetes Benchmark 1.4.1 - Rancher 2.3.0-2.3.2 with Kubernetes 1.15

 [Click here to download a PDF version of this document](https://releases.rancher.com/documents/security/2.3.x/Rancher_Benchmark_Assessment.pdf)

@@ -445,6 +445,7 @@ docker inspect kube-apiserver | jq -e '.[0].Args[] | match("--client-ca-file=.*"
 ```

 **Returned Value:** `--client-ca-file=/etc/kubernetes/ssl/kube-ca.pem`
+
 **Result:** Pass

 #### 1.1.30 - Ensure that the API Server only makes use of strong cryptographic ciphers (Not Scored)
@@ -536,6 +537,7 @@ docker inspect kube-apiserver | jq -e '.[0].Args[] | match("--authorization-mode
 ```

 **Returned Value:** `--authorization-mode=Node,RBAC`
+
 **Result:** Pass

 #### 1.1.33 - Ensure that the admission control plugin `NodeRestriction` is set (Scored)
@@ -673,6 +675,7 @@ docker inspect kube-scheduler | jq -e '.[0].Args[] | match("--profiling=false").
 ```

 **Returned Value:** `--profiling=false`
+
 **Result:** Pass

 #### 1.2.2 - Ensure that the `--address` argument is set to 127.0.0.1 (Scored)
@@ -684,6 +687,7 @@ docker inspect kube-scheduler | jq -e '.[0].Args[] | match("--address=127\\.0\\.
 ```

 **Returned Value:** `--address=127.0.0.1`
+
 **Result:** Pass

 ### 1.3 - Controller Manager
@@ -39,6 +39,8 @@ Items in this profile extend the “Level 1” profile and exhibit one or more o

 ## 1.1 - Rancher HA Kubernetes cluster host configuration

+(See Appendix A. for full ubuntu `cloud-config` example)
+
 ### 1.1.1 - Configure default sysctl settings on all hosts

 **Profile Applicability**
@@ -65,6 +67,12 @@ This supports the following control:
 sysctl vm.overcommit_memory
 ```

+- Verify `vm.panic_on_oom = 0`
+
+``` bash
+sysctl vm.panic_on_oom
+```
+
 - Verify `kernel.panic = 10`

 ``` bash
@@ -77,17 +85,32 @@ sysctl kernel.panic
 sysctl kernel.panic_on_oops
 ```

+- Verify `kernel.keys.root_maxkeys = 1000000`
+
+``` bash
+sysctl kernel.keys.root_maxkeys
+```
+
+- Verify `kernel.keys.root_maxbytes = 25000000`
+
+``` bash
+sysctl kernel.keys.root_maxbytes
+```
+
 **Remediation**

- Set the following parameters in `/etc/sysctl.conf` on all nodes:
+- Set the following parameters in `/etc/sysctl.d/90-kubelet.conf` on all nodes:

 ``` plain
 vm.overcommit_memory=1
+vm.panic_on_oom=0
 kernel.panic=10
 kernel.panic_on_oops=1
+kernel.keys.root_maxkeys=1000000
+kernel.keys.root_maxbytes=25000000
 ```

- Run `sysctl -p` to enable the settings.
+- Run `sysctl -p /etc/sysctl.d/90-kubelet.conf` to enable the settings.

 ### 1.1.2 - Install the encryption provider configuration on all control plane nodes

@@ -442,7 +465,7 @@ services:

 ## 2.1 - Rancher HA Kubernetes Cluster Configuration via RKE

-(See Appendix A. for full RKE `cluster.yml` example)
+(See Appendix B. for full RKE `cluster.yml` example)

 ### 2.1.1 - Configure kubelet options

@@ -1122,7 +1145,97 @@ If a disallowed node driver is active, visit the _Node Drivers_ page under _Glob

 ---

-## Appendix A - Complete RKE `cluster.yml` Example
+## Appendix A - Complete ubuntu `cloud-config` Example
+
+`cloud-config` file to automate hardening manual steps on nodes deployment.
+
+```
+#cloud-config
+bootcmd:
+- apt-get update
+- apt-get install -y apt-transport-https
+apt:
+  sources:
+    docker:
+      source: "deb [arch=amd64] https://download.docker.com/linux/ubuntu $RELEASE stable"
+      keyid: 0EBFCD88
+packages:
+- [docker-ce, '5:19.03.5~3-0~ubuntu-bionic']
+- jq
+write_files:
+# 1.1.1 - Configure default sysctl settings on all hosts
+- path: /etc/sysctl.d/90-kubelet.conf
+  owner: root:root
+  permissions: '0644'
+  content: |
+    vm.overcommit_memory=1
+    vm.panic_on_oom=0
+    kernel.panic=10
+    kernel.panic_on_oops=1
+    kernel.keys.root_maxkeys=1000000
+    kernel.keys.root_maxbytes=25000000
+# 1.1.2 encription provider
+- path: /opt/kubernetes/encryption.yaml
+  owner: root:root
+  permissions: '0600'
+  content: |
+    apiVersion: apiserver.config.k8s.io/v1
+    kind: EncryptionConfiguration
+    resources:
+      - resources:
+        - secrets
+        providers:
+        - aescbc:
+            keys:
+            - name: key1
+              secret: QRCexFindur3dzS0P/UmHs5xA6sKu58RbtWOQFarfh4=
+        - identity: {}
+# 1.1.3 audit log
+- path: /opt/kubernetes/audit.yaml
+  owner: root:root
+  permissions: '0600'
+  content: |
+    apiVersion: audit.k8s.io/v1beta1
+    kind: Policy
+    rules:
+    - level: Metadata
+# 1.1.4 event limit
+- path: /opt/kubernetes/admission.yaml
+  owner: root:root
+  permissions: '0600'
+  content: |
+    apiVersion: apiserver.k8s.io/v1alpha1
+    kind: AdmissionConfiguration
+    plugins:
+    - name: EventRateLimit
+      path: /opt/kubernetes/event.yaml
+- path: /opt/kubernetes/event.yaml
+  owner: root:root
+  permissions: '0600'
+  content: |
+    apiVersion: eventratelimit.admission.k8s.io/v1alpha1
+    kind: Configuration
+    limits:
+    - type: Server
+      qps: 5000
+      burst: 20000
+# 1.4.12 etcd user
+groups:
+  - etcd
+users:
+  - default
+  - name: etcd
+    gecos: Etcd user
+    primary_group: etcd
+    homedir: /var/lib/etcd
+# 1.4.11 etcd data dir
+runcmd:
+  - chmod 0700 /var/lib/etcd
+  - usermod -G docker -a ubuntu
+  - sysctl -p /etc/sysctl.d/90-kubelet.conf
+```
+
+## Appendix B - Complete RKE `cluster.yml` Example

 ``` yaml
 nodes:
@@ -1315,7 +1428,7 @@ addons: |
    name: system:authenticated
 ```

-## Appendix B - Complete RKE Template Example
+## Appendix C - Complete RKE Template Example

 ``` yaml
 #