Merge pull request #2023 from rawmind0/master

Updated hardening guide to build RKE clusters CIS compliant

content/rancher/v2.x/en/security/_index.md

@@ -49,7 +49,8 @@ The Rancher Hardening Guide is based off of controls and best practices found in
 
 - [Hardening Guide for Rancher v2.1.x with Kubernetes 1.11]({{< baseurl >}}/rancher/v2.x/en/security/hardening-2.1/)
 - [Hardening Guide for Rancher v2.2.x with Kubernetes 1.13]({{< baseurl >}}/rancher/v2.x/en/security/hardening-2.2/)
-- [Hardening Guide for Rancher v2.3.x with Kubernetes 1.15]({{< baseurl >}}/rancher/v2.x/en/security/hardening-2.3/)
+- [Hardening Guide for Rancher v2.3.0-v2.3.2 with Kubernetes 1.15]({{< baseurl >}}/rancher/v2.x/en/security/hardening-2.3/)
+- [Hardening Guide for Rancher v2.3.3+ with Kubernetes 1.16]({{< baseurl >}}/rancher/v2.x/en/security/hardening-2.3.3/)
 
 ### The CIS Benchmark and Self-Assessment
 
@@ -57,10 +58,11 @@ The benchmark self-assessment is a companion to the Rancher security hardening g
 
 Because Rancher and RKE install Kubernetes services as Docker containers, many of the control verification checks in the CIS Kubernetes Benchmark don't apply. This guide will walk through the various controls and provide updated example commands to audit compliance in Rancher created clusters.  The original benchmark documents can be downloaded from the [CIS website](https://www.cisecurity.org/benchmark/kubernetes/).
 
-- [CIS Kubernetes Benchmark 1.3.0 - Rancher 2.1.x with Kubernetes 1.11]({{< baseurl >}}/rancher/v2.x/en/security/benchmark-2.1/)
+* [CIS Kubernetes Benchmark 1.3.0 - Rancher 2.1.x with Kubernetes 1.11]({{< baseurl >}}/rancher/v2.x/en/security/benchmark-2.1/)
-- [CIS Kubernetes Benchmark 1.4.0 - Rancher 2.2.x with Kubernetes 1.13]({{< baseurl >}}/rancher/v2.x/en/security/benchmark-2.2/#cis-kubernetes-benchmark-1-4-0-rancher-2-2-x-with-kubernetes-1-13/)
+* [CIS Kubernetes Benchmark 1.4.0 - Rancher 2.2.x with Kubernetes 1.13]({{< baseurl >}}/rancher/v2.x/en/security/benchmark-2.2/#cis-kubernetes-benchmark-1-4-0-rancher-2-2-x-with-kubernetes-1-13/)
-- [CIS Kubernetes Benchmark 1.4.1 - Rancher 2.2.x with Kubernetes 1.13]({{< baseurl >}}/rancher/v2.x/en/security/benchmark-2.2/#cis-kubernetes-benchmark-1-4-1-rancher-2-2-x-with-kubernetes-1-13)
+* [CIS Kubernetes Benchmark 1.4.1 - Rancher 2.2.x with Kubernetes 1.13]({{< baseurl >}}/rancher/v2.x/en/security/benchmark-2.2/#cis-kubernetes-benchmark-1-4-1-rancher-2-2-x-with-kubernetes-1-13)
-- [CIS Kubernetes Benchmark 1.4.1 - Rancher 2.3.x with Kubernetes 1.15]({{< baseurl >}}/rancher/v2.x/en/security/benchmark-2.3/#cis-kubernetes-benchmark-1-4-1-rancher-2-3-x-with-kubernetes-1-15)
+* [CIS Kubernetes Benchmark 1.4.1 - Rancher 2.3.0-2.3.2 with Kubernetes 1.15]({{< baseurl >}}/rancher/v2.x/en/security/benchmark-2.3/#cis-kubernetes-benchmark-1-4-1-rancher-2-3-0-2-3-2-with-kubernetes-1-15)
+* [CIS Kubernetes Benchmark 1.4.1 - Rancher 2.3.3+ with Kubernetes 1.16]({{< baseurl >}}/rancher/v2.x/en/security/benchmark-2.3.3/#cis-kubernetes-benchmark-1-4-1-rancher-2-3-3-+-with-kubernetes-1-16)
 
 ### Third-party Penetration Test Reports
 

content/rancher/v2.x/en/security/benchmark-2.3/_index.md

@@ -1,9 +1,9 @@
 ---
-title: CIS Benchmark Rancher Self-Assessment Guide - Rancher v2.3.x
+title: CIS Benchmark Rancher Self-Assessment Guide - Rancher v2.3.0-v2.3.2
 weight: 103
 ---
 
-### CIS Kubernetes Benchmark 1.4.1 - Rancher 2.3.x with Kubernetes 1.15
+### CIS Kubernetes Benchmark 1.4.1 - Rancher 2.3.0-2.3.2 with Kubernetes 1.15
 
 [Click here to download a PDF version of this document](https://releases.rancher.com/documents/security/2.3.x/Rancher_Benchmark_Assessment.pdf)
 
@@ -445,6 +445,7 @@ docker inspect kube-apiserver | jq -e '.[0].Args[] | match("--client-ca-file=.*"
 ```
 
 **Returned Value:** `--client-ca-file=/etc/kubernetes/ssl/kube-ca.pem`
+
 **Result:** Pass
 
 #### 1.1.30 - Ensure that the API Server only makes use of strong cryptographic ciphers (Not Scored)
@@ -536,6 +537,7 @@ docker inspect kube-apiserver | jq -e '.[0].Args[] | match("--authorization-mode
 ```
 
 **Returned Value:** `--authorization-mode=Node,RBAC`
+
 **Result:** Pass
 
 #### 1.1.33 - Ensure that the admission control plugin `NodeRestriction` is set (Scored)
@@ -673,6 +675,7 @@ docker inspect kube-scheduler | jq -e '.[0].Args[] | match("--profiling=false").
 ```
 
 **Returned Value:** `--profiling=false`
+
 **Result:** Pass
 
 #### 1.2.2 - Ensure that the `--address` argument is set to 127.0.0.1 (Scored)
@@ -684,6 +687,7 @@ docker inspect kube-scheduler | jq -e '.[0].Args[] | match("--address=127\\.0\\.
 ```
 
 **Returned Value:** `--address=127.0.0.1`
+
 **Result:** Pass
 
 ### 1.3 - Controller Manager

content/rancher/v2.x/en/security/hardening-2.3/_index.md

@@ -39,6 +39,8 @@ Items in this profile extend the “Level 1” profile and exhibit one or more o
 
 ## 1.1 - Rancher HA Kubernetes cluster host configuration
 
+(See Appendix A. for full ubuntu `cloud-config` example)
+
 ### 1.1.1 - Configure default sysctl settings on all hosts
 
 **Profile Applicability**
@@ -65,6 +67,12 @@ This supports the following control:
 sysctl vm.overcommit_memory
 ```
 
+- Verify `vm.panic_on_oom = 0`
+
+``` bash
+sysctl vm.panic_on_oom
+```
+
 - Verify `kernel.panic = 10`
 
 ``` bash
@@ -77,17 +85,32 @@ sysctl kernel.panic
 sysctl kernel.panic_on_oops
 ```
 
+- Verify `kernel.keys.root_maxkeys = 1000000`
+
+``` bash
+sysctl kernel.keys.root_maxkeys
+```
+
+- Verify `kernel.keys.root_maxbytes = 25000000`
+
+``` bash
+sysctl kernel.keys.root_maxbytes
+```
+
 **Remediation**
 
-- Set the following parameters in `/etc/sysctl.conf` on all nodes:
+- Set the following parameters in `/etc/sysctl.d/90-kubelet.conf` on all nodes:
 
 ``` plain
 vm.overcommit_memory=1
+vm.panic_on_oom=0
 kernel.panic=10
 kernel.panic_on_oops=1
+kernel.keys.root_maxkeys=1000000
+kernel.keys.root_maxbytes=25000000
 ```
 
-- Run `sysctl -p` to enable the settings.
+- Run `sysctl -p /etc/sysctl.d/90-kubelet.conf` to enable the settings.
 
 ### 1.1.2 - Install the encryption provider configuration on all control plane nodes
 
@@ -442,7 +465,7 @@ services:
 
 ## 2.1 - Rancher HA Kubernetes Cluster Configuration via RKE
 
-(See Appendix A. for full RKE `cluster.yml` example)
+(See Appendix B. for full RKE `cluster.yml` example)
 
 ### 2.1.1 - Configure kubelet options
 
@@ -1122,7 +1145,97 @@ If a disallowed node driver is active, visit the _Node Drivers_ page under _Glob
 
 ---
 
-## Appendix A - Complete RKE `cluster.yml` Example
+## Appendix A - Complete ubuntu `cloud-config` Example
+
+`cloud-config` file to automate hardening manual steps on nodes deployment.
+
+```
+#cloud-config
+bootcmd:
+- apt-get update
+- apt-get install -y apt-transport-https
+apt:
+  sources:
+    docker:
+      source: "deb [arch=amd64] https://download.docker.com/linux/ubuntu $RELEASE stable"
+      keyid: 0EBFCD88
+packages:
+- [docker-ce, '5:19.03.5~3-0~ubuntu-bionic']
+- jq
+write_files:
+# 1.1.1 - Configure default sysctl settings on all hosts
+- path: /etc/sysctl.d/90-kubelet.conf
+  owner: root:root
+  permissions: '0644'
+  content: |
+    vm.overcommit_memory=1
+    vm.panic_on_oom=0
+    kernel.panic=10
+    kernel.panic_on_oops=1
+    kernel.keys.root_maxkeys=1000000
+    kernel.keys.root_maxbytes=25000000
+# 1.1.2 encription provider
+- path: /opt/kubernetes/encryption.yaml
+  owner: root:root
+  permissions: '0600'
+  content: |
+    apiVersion: apiserver.config.k8s.io/v1
+    kind: EncryptionConfiguration
+    resources:
+      - resources:
+        - secrets
+        providers:
+        - aescbc:
+            keys:
+            - name: key1
+              secret: QRCexFindur3dzS0P/UmHs5xA6sKu58RbtWOQFarfh4=
+        - identity: {}
+# 1.1.3 audit log
+- path: /opt/kubernetes/audit.yaml
+  owner: root:root
+  permissions: '0600'
+  content: |
+    apiVersion: audit.k8s.io/v1beta1
+    kind: Policy
+    rules:
+    - level: Metadata
+# 1.1.4 event limit
+- path: /opt/kubernetes/admission.yaml
+  owner: root:root
+  permissions: '0600'
+  content: |
+    apiVersion: apiserver.k8s.io/v1alpha1
+    kind: AdmissionConfiguration
+    plugins:
+    - name: EventRateLimit
+      path: /opt/kubernetes/event.yaml
+- path: /opt/kubernetes/event.yaml
+  owner: root:root
+  permissions: '0600'
+  content: |
+    apiVersion: eventratelimit.admission.k8s.io/v1alpha1
+    kind: Configuration
+    limits:
+    - type: Server
+      qps: 5000
+      burst: 20000
+# 1.4.12 etcd user
+groups:
+  - etcd
+users:
+  - default
+  - name: etcd
+    gecos: Etcd user
+    primary_group: etcd
+    homedir: /var/lib/etcd
+# 1.4.11 etcd data dir
+runcmd:
+  - chmod 0700 /var/lib/etcd
+  - usermod -G docker -a ubuntu
+  - sysctl -p /etc/sysctl.d/90-kubelet.conf
+```
+
+## Appendix B - Complete RKE `cluster.yml` Example
 
 ``` yaml
 nodes:
@@ -1315,7 +1428,7 @@ addons: |
     name: system:authenticated
 ```
 
-## Appendix B - Complete RKE Template Example
+## Appendix C - Complete RKE Template Example
 
 ``` yaml
 #