From f76de8a848fe0f28f484b49dafb2ef54c7c0284c Mon Sep 17 00:00:00 2001 From: Nelson Roberts Date: Wed, 29 Jan 2020 11:03:11 -0700 Subject: [PATCH 01/14] EIO-247: security docs for 2.3.4 --- .../en/security/benchmark-2.3.4/_index.md | 2310 +++++++++++++++++ .../en/security/hardening-2.3.4/_index.md | 389 +++ 2 files changed, 2699 insertions(+) create mode 100644 content/rancher/v2.x/en/security/benchmark-2.3.4/_index.md create mode 100644 content/rancher/v2.x/en/security/hardening-2.3.4/_index.md diff --git a/content/rancher/v2.x/en/security/benchmark-2.3.4/_index.md b/content/rancher/v2.x/en/security/benchmark-2.3.4/_index.md new file mode 100644 index 00000000000..669565d9a82 --- /dev/null +++ b/content/rancher/v2.x/en/security/benchmark-2.3.4/_index.md @@ -0,0 +1,2310 @@ +--- +title: CIS Benchmark Rancher Self-Assessment Guide - Rancher v2.3.4 +weight: 103 +--- + +### CIS Kubernetes Benchmark 1.5 - Rancher 2.3.4 with Kubernetes 1.15 + +[Click here to download a PDF version of this document](https://releases.rancher.com/documents/security/2.3.4/Rancher_Benchmark_Assessment.pdf) + +#### Overview + +The following document scores a Kubernetes 1.15.x RKE cluster provisioned according to the Rancher v2.3.4 hardening guide against the CIS 1.5 Kubernetes benchmark. + +This document is a companion to the Rancher v2.3.4 security hardening guide. The hardening guide provides prescriptive guidance for hardening a production installation of Rancher, and this benchmark guide is meant to help you evaluate the level of security of the hardened cluster against each control in the benchmark. + +Because Rancher and RKE install Kubernetes services as Docker containers, many of the control verification checks in the CIS Kubernetes Benchmark don't apply. This guide will walk through the various controls and provide updated example commands to audit compliance in Rancher-created clusters. + +This document is to be used by Rancher operators, security teams, auditors and decision makers. + +For more detail about each audit, including rationales and remediations for failing tests, you can refer to the corresponding section of the CIS Kubernetes Benchmark v1.5. You can download the benchmark after logging in to [CISecurity.org]( https://www.cisecurity.org/benchmark/kubernetes/). + +#### Testing controls methodology + +Rancher and RKE install Kubernetes services via Docker containers. Configuration is defined by arguments passed to the container at the time of initialization, not via configuration files. + +Scoring the commands is different in Rancher Labs than in the CIS Benchmark. Where the commands differ from the original CIS benchmark, the commands specific to Rancher Labs are provided for testing. + +When performing the tests, you will need access to the Docker command line on the hosts of all three RKE roles. The commands also make use of the the `jq` command to provide human-readable formatting. + +#### Known Scored Control Failures + +The following scored controls do not currently pass, and Rancher Labs is working towards addressing these through future enhancements to the product. + +- 1.1.11 Ensure that the etcd data directory permissions are set to 700 or more restrictive (Scored) +- 1.1.12 Ensure that the etcd data directory ownership is set to etcd:etcd (Scored) +- 1.2.6 Ensure that the --kubelet-certificate-authority argument is set as appropriate (Scored) + +### Controls + +--- +## 1 Master Node Security Configuration +### 1.1 Master Node Configuration Files + +#### 1.1.1 Ensure that the API server pod specification file permissions are set to `644` or more restrictive (Scored) + +**Result:** INFO + +**Remediation:** +RKE doesn’t require or maintain a configuration file for the API server. All configuration is passed in as arguments at container run time. + +#### 1.1.2 Ensure that the API server pod specification file ownership is set to `root:root` (Scored) + +**Result:** INFO + +**Remediation:** +RKE doesn’t require or maintain a configuration file for the API server. All configuration is passed in as arguments at container run time. + +#### 1.1.3 Ensure that the controller manager pod specification file permissions are set to `644` or more restrictive (Scored) + +**Result:** INFO + +**Remediation:** +RKE doesn’t require or maintain a configuration file for the controller manager. All configuration is passed in as arguments at container run time. + +#### 1.1.4 Ensure that the controller manager pod specification file ownership is set to `root:root` (Scored) + +**Result:** INFO + +**Remediation:** +RKE doesn’t require or maintain a configuration file for the controller manager. All configuration is passed in as arguments at container run time. + +#### 1.1.5 Ensure that the scheduler pod specification file permissions are set to `644` or more restrictive (Scored) + +**Result:** INFO + +**Remediation:** +RKE doesn’t require or maintain a configuration file for the scheduler. All configuration is passed in as arguments at container run time. + +#### 1.1.6 Ensure that the scheduler pod specification file ownership is set to `root:root` (Scored) + +**Result:** INFO + +**Remediation:** +RKE doesn’t require or maintain a configuration file for the scheduler. All configuration is passed in as arguments at container run time. + +#### 1.1.7 Ensure that the etcd pod specification file permissions are set to `644` or more restrictive (Scored) + +**Result:** INFO + +**Remediation:** +RKE doesn’t require or maintain a configuration file for etcd. All configuration is passed in as arguments at container run time. + +#### 1.1.8 Ensure that the etcd pod specification file ownership is set to `root:root` (Scored) + +**Result:** INFO + +**Remediation:** +RKE doesn’t require or maintain a configuration file for etcd. All configuration is passed in as arguments at container run time. + +#### 1.1.9 Ensure that the Container Network Interface file permissions are set to `644` or more restrictive (Not Scored) + +**Result:** WARN + +**Remediation:** +Run the below command (based on the file location on your system) on the master node. +For example, + +``` bash +chmod 644 +``` + +**Audit:** + +``` bash +stat -c %a +``` + +#### 1.1.10 Ensure that the Container Network Interface file ownership is set to `root:root` (Not Scored) + +**Result:** WARN + +**Remediation:** +Run the below command (based on the file location on your system) on the master node. +For example, + +``` bash +chown root:root +``` + +**Audit:** + +``` bash +stat -c %U:%G +``` + +#### 1.1.11 Ensure that the etcd data directory permissions are set to `700` or more restrictive (Scored) + +**Result:** FAIL + +**Remediation:** +On the etcd server node, get the etcd data directory, passed as an argument `--data-dir`, +from the below command: + +``` bash +ps -ef | grep etcd +``` + +Run the below command (based on the etcd data directory found above). For example, + +``` bash +chmod 700 /var/lib/etcd +``` + +**Audit:** + +``` bash +ps -ef | grep etcd | grep -- --data-dir | sed 's%.*data-dir[= ]\([^ ]*\).*%\1%' | xargs stat -c %a +``` + +#### 1.1.12 Ensure that the etcd data directory ownership is set to `etcd:etcd` (Scored) + +**Result:** FAIL + +**Remediation:** +On the etcd server node, get the etcd data directory, passed as an argument `--data-dir`, +from the below command: + +``` bash +ps -ef | grep etcd +``` + +Run the below command (based on the etcd data directory found above). +For example, +``` bash +chown etcd:etcd /var/lib/etcd +``` + +**Audit:** + +``` bash +ps -ef | grep etcd | grep -- --data-dir | sed 's%.*data-dir[= ]\([^ ]*\).*%\1%' | xargs stat -c %U:%G +``` + +#### 1.1.13 Ensure that the `admin.conf` file permissions are set to `644` or more restrictive (Scored) + +**Result:** INFO + +**Remediation:** +RKE does not store the kubernetes default kubeconfig credentials file on the nodes. It’s presented to user where RKE is run. +We recommend that this `kube_config_cluster.yml` file be kept in secure store. + +#### 1.1.14 Ensure that the admin.conf file ownership is set to `root:root` (Scored) + +**Result:** INFO + +**Remediation:** +RKE does not store the kubernetes default kubeconfig credentials file on the nodes. It’s presented to user where RKE is run. +We recommend that this `kube_config_cluster.yml` file be kept in secure store. + +#### 1.1.15 Ensure that the `scheduler.conf` file permissions are set to `644` or more restrictive (Scored) + +**Result:** INFO + +**Remediation:** +RKE doesn’t require or maintain a configuration file for the scheduler. All configuration is passed in as arguments at container run time. + +#### 1.1.16 Ensure that the `scheduler.conf` file ownership is set to `root:root` (Scored) + +**Result:** INFO + +**Remediation:** +RKE doesn’t require or maintain a configuration file for the scheduler. All configuration is passed in as arguments at container run time. + +#### 1.1.17 Ensure that the `controller-manager.conf` file permissions are set to `644` or more restrictive (Scored) + +**Result:** INFO + +**Remediation:** +RKE doesn’t require or maintain a configuration file for the controller manager. All configuration is passed in as arguments at container run time. + +#### 1.1.18 Ensure that the `controller-manager.conf` file ownership is set to `root:root` (Scored) + +**Result:** INFO + +**Remediation:** +RKE doesn’t require or maintain a configuration file for the controller manager. All configuration is passed in as arguments at container run time. + +#### 1.1.19 Ensure that the Kubernetes PKI directory and file ownership is set to `root:root` (Scored) + +**Result:** WARN + +**Remediation:** +Run the below command (based on the file location on your system) on the master node. +For example, + +``` bash +chown -R root:root /etc/kubernetes/pki/ +``` + +**Audit:** + +``` bash +ls -laR /etc/kubernetes/pki/ +``` + +#### 1.1.20 Ensure that the Kubernetes PKI certificate file permissions are set to `644` or more restrictive (Scored) + +**Result:** WARN + +**Remediation:** +Run the below command (based on the file location on your system) on the master node. +For example, + +``` bash +chmod -R 644 /etc/kubernetes/pki/*.crt +``` + +**Audit:** + +``` bash +stat -c %n %a /etc/kubernetes/pki/*.crt +``` + +#### 1.1.21 Ensure that the Kubernetes PKI key file permissions are set to `600` (Scored) + +**Result:** WARN + +**Remediation:** +Run the below command (based on the file location on your system) on the master node. +For example, + +``` bash +chmod -R 600 /etc/kubernetes/pki/*.key +``` + +**Audit:** + +``` bash +stat -c %n %a /etc/kubernetes/pki/*.key +``` + +### 1.2 API Server + +#### 1.2.1 Ensure that the `--anonymous-auth` argument is set to `false` (Not Scored) + +**Result:** PASS + +**Remediation:** +Edit the API server pod specification file `/etc/kubernetes/manifests/kube-apiserver.yaml` +on the master node and set the below parameter. + +``` bash +--anonymous-auth=false +``` + +**Audit:** + +``` bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected result**: + +``` bash +'false' is equal to 'false' +``` + +#### 1.2.2 Ensure that the `--basic-auth-file` argument is not set (Scored) + +**Result:** PASS + +**Remediation:** +Follow the documentation and configure alternate mechanisms for authentication. Then, +edit the API server pod specification file `/etc/kubernetes/manifests/kube-apiserver.yaml` +on the master node and remove the `--basic-auth-file=` parameter. + +**Audit:** + +``` bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected result**: + +``` bash +'--basic-auth-file' is not present +``` + +#### 1.2.3 Ensure that the `--token-auth-file` parameter is not set (Scored) + +**Result:** PASS + +**Remediation:** +Follow the documentation and configure alternate mechanisms for authentication. Then, +edit the API server pod specification file `/etc/kubernetes/manifests/kube-apiserver.yaml` +on the master node and remove the `--token-auth-file=` parameter. + +**Audit:** + +``` bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected result**: + +``` bash +'--token-auth-file' is not present +``` + +#### 1.2.4 Ensure that the `--kubelet-https` argument is set to true (Scored) + +**Result:** PASS + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the master node and remove the `--kubelet-https` parameter. + +**Audit:** + +``` bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected result**: + +``` bash +'--kubelet-https' is present OR '--kubelet-https' is not present +``` + +#### 1.2.5 Ensure that the `--kubelet-client-certificate` and `--kubelet-client-key` arguments are set as appropriate (Scored) + +**Result:** PASS + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection between the +apiserver and kubelets. Then, edit API server pod specification file +`/etc/kubernetes/manifests/kube-apiserver.yaml` on the master node and set the +kubelet client certificate and key parameters as below. + +``` bash +--kubelet-client-certificate= +--kubelet-client-key= +``` + +**Audit:** + +``` bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected result**: + +``` bash +'--kubelet-client-certificate' is present AND '--kubelet-client-key' is present +``` + +#### 1.2.6 Ensure that the `--kubelet-certificate-authority` argument is set as appropriate (Scored) + +**Result:** FAIL + +**Remediation:** +Follow the Kubernetes documentation and setup the TLS connection between +the apiserver and kubelets. Then, edit the API server pod specification file +`/etc/kubernetes/manifests/kube-apiserver.yaml` on the master node and set the +`--kubelet-certificate-authority` parameter to the path to the cert file for the certificate authority. +`--kubelet-certificate-authority=` + +**Audit:** + +``` bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +#### 1.2.7 Ensure that the `--authorization-mode` argument is not set to `AlwaysAllow` (Scored) + +**Result:** PASS + +**Remediation:** +Edit the API server pod specification file `/etc/kubernetes/manifests/kube-apiserver.yaml` +on the master node and set the `--authorization-mode` parameter to values other than `AlwaysAllow`. +One such example could be as below. + +``` bash +--authorization-mode=RBAC +``` + +**Audit:** + +``` bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected result**: + +``` bash +'Node,RBAC' not have 'AlwaysAllow' +``` + +#### 1.2.8 Ensure that the `--authorization-mode` argument includes `Node` (Scored) + +**Result:** PASS + +**Remediation:** +Edit the API server pod specification file `/etc/kubernetes/manifests/kube-apiserver.yaml` +on the master node and set the `--authorization-mode` parameter to a value that includes `Node`. + +``` bash +--authorization-mode=Node,RBAC +``` + +**Audit:** + +``` bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected result**: + +``` bash +'Node,RBAC' has 'Node' +``` + +#### 1.2.9 Ensure that the `--authorization-mode` argument includes `RBAC` (Scored) + +**Result:** PASS + +**Remediation:** +Edit the API server pod specification file `/etc/kubernetes/manifests/kube-apiserver.yaml` +on the master node and set the `--authorization-mode` parameter to a value that includes RBAC, +for example: + +``` bash +--authorization-mode=Node,RBAC +``` + +**Audit:** + +``` bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected result**: + +``` bash +'Node,RBAC' has 'RBAC' +``` + +#### 1.2.10 Ensure that the admission control plugin `EventRateLimit` is set (Not Scored) + +**Result:** PASS + +**Remediation:** +Follow the Kubernetes documentation and set the desired limits in a configuration file. +Then, edit the API server pod specification file `/etc/kubernetes/manifests/kube-apiserver.yaml` +and set the below parameters. + +``` bash +--enable-admission-plugins=...,EventRateLimit,... +--admission-control-config-file= +``` + +**Audit:** + +``` bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected result**: + +``` bash +'ServiceAccount,NamespaceLifecycle,LimitRanger,PersistentVolumeLabel,DefaultStorageClass,ResourceQuota,DefaultTolerationSeconds,AlwaysPullImages,DenyEscalatingExec,NodeRestriction,EventRateLimit,PodSecurityPolicy' has 'EventRateLimit' +``` + +#### 1.2.11 Ensure that the admission control plugin `AlwaysAdmit` is not set (Scored) + +**Result:** PASS + +**Remediation:** +Edit the API server pod specification file `/etc/kubernetes/manifests/kube-apiserver.yaml` +on the master node and either remove the `--enable-admission-plugins` parameter, or set it to a +value that does not include `AlwaysAdmit`. + +**Audit:** + +``` bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected result**: + +``` bash +'ServiceAccount,NamespaceLifecycle,LimitRanger,PersistentVolumeLabel,DefaultStorageClass,ResourceQuota,DefaultTolerationSeconds,AlwaysPullImages,DenyEscalatingExec,NodeRestriction,EventRateLimit,PodSecurityPolicy' not have 'AlwaysAdmit' OR '--enable-admission-plugins' is not present +``` + +#### 1.2.12 Ensure that the admission control plugin `AlwaysPullImages` is set (Not Scored) + +**Result:** PASS + +**Remediation:** +Edit the API server pod specification file `/etc/kubernetes/manifests/kube-apiserver.yaml` +on the master node and set the `--enable-admission-plugins` parameter to include +`AlwaysPullImages`. + +``` bash +--enable-admission-plugins=...,AlwaysPullImages,... +``` + +**Audit:** + +``` bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected result**: + +``` bash +'ServiceAccount,NamespaceLifecycle,LimitRanger,PersistentVolumeLabel,DefaultStorageClass,ResourceQuota,DefaultTolerationSeconds,AlwaysPullImages,DenyEscalatingExec,NodeRestriction,EventRateLimit,PodSecurityPolicy' has 'AlwaysPullImages' +``` + +#### 1.2.13 Ensure that the admission control plugin `SecurityContextDeny` is set if `PodSecurityPolicy` is not used (Not Scored) + +**Result:** WARN + +**Remediation:** +Edit the API server pod specification file `/etc/kubernetes/manifests/kube-apiserver.yaml` +on the master node and set the `--enable-admission-plugins` parameter to include +`SecurityContextDeny`, unless `PodSecurityPolicy` is already in place. + +``` bash +--enable-admission-plugins=...,SecurityContextDeny,... +``` + +**Audit:** + +``` bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +#### 1.2.14 Ensure that the admission control plugin `ServiceAccount` is set (Scored) + +**Result:** PASS + +**Remediation:** +Follow the documentation and create ServiceAccount objects as per your environment. +Then, edit the API server pod specification file `/etc/kubernetes/manifests/kube-apiserver.yaml` +on the master node and ensure that the `--disable-admission-plugins` parameter is set to a +value that does not include `ServiceAccount`. + +**Audit:** + +``` bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected result**: + +``` bash +'ServiceAccount,NamespaceLifecycle,LimitRanger,PersistentVolumeLabel,DefaultStorageClass,ResourceQuota,DefaultTolerationSeconds,AlwaysPullImages,DenyEscalatingExec,NodeRestriction,EventRateLimit,PodSecurityPolicy' has 'ServiceAccount' OR '--enable-admission-plugins' is not present +``` + +#### 1.2.15 Ensure that the admission control plugin `NamespaceLifecycle` is set (Scored) + +**Result:** PASS + +**Remediation:** +Edit the API server pod specification file `/etc/kubernetes/manifests/kube-apiserver.yaml` +on the master node and set the `--disable-admission-plugins` parameter to +ensure it does not include `NamespaceLifecycle`. + +**Audit:** + +``` bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected result**: + +``` bash +'--disable-admission-plugins' is present OR '--disable-admission-plugins' is not present +``` + +#### 1.2.16 Ensure that the admission control plugin `PodSecurityPolicy` is set (Scored) + +**Result:** PASS + +**Remediation:** +Follow the documentation and create Pod Security Policy objects as per your environment. +Then, edit the API server pod specification file `/etc/kubernetes/manifests/kube-apiserver.yaml` +on the master node and set the `--enable-admission-plugins` parameter to a +value that includes `PodSecurityPolicy`: + +``` bash +--enable-admission-plugins=...,PodSecurityPolicy,... +``` + +Then restart the API Server. + +**Audit:** + +``` bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected result**: + +``` bash +'ServiceAccount,NamespaceLifecycle,LimitRanger,PersistentVolumeLabel,DefaultStorageClass,ResourceQuota,DefaultTolerationSeconds,AlwaysPullImages,DenyEscalatingExec,NodeRestriction,EventRateLimit,PodSecurityPolicy' has 'PodSecurityPolicy' +``` + +#### 1.2.17 Ensure that the admission control plugin `NodeRestriction` is set (Scored) + +**Result:** PASS + +**Remediation:** +Follow the Kubernetes documentation and configure `NodeRestriction` plug-in on kubelets. +Then, edit the API server pod specification file `/etc/kubernetes/manifests/kube-apiserver.yaml` +on the master node and set the `--enable-admission-plugins` parameter to a +value that includes `NodeRestriction`. + +``` bash +--enable-admission-plugins=...,NodeRestriction,... +``` + +**Audit:** + +``` bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected result**: + +``` bash +'ServiceAccount,NamespaceLifecycle,LimitRanger,PersistentVolumeLabel,DefaultStorageClass,ResourceQuota,DefaultTolerationSeconds,AlwaysPullImages,DenyEscalatingExec,NodeRestriction,EventRateLimit,PodSecurityPolicy' has 'NodeRestriction' +``` + +#### 1.2.18 Ensure that the `--insecure-bind-address` argument is not set (Scored) + +**Result:** PASS + +**Remediation:** +Edit the API server pod specification file `/etc/kubernetes/manifests/kube-apiserver.yaml` +on the master node and remove the `--insecure-bind-address` parameter. + +**Audit:** + +``` bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected result**: + +``` bash +'--insecure-bind-address' is not present +``` + +#### 1.2.19 Ensure that the `--insecure-port` argument is set to `0` (Scored) + +**Result:** PASS + +**Remediation:** +Edit the API server pod specification file `/etc/kubernetes/manifests/kube-apiserver.yaml` +on the master node and set the below parameter. + +``` bash +--insecure-port=0 +``` + +**Audit:** + +``` bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected result**: + +``` bash +'0' is equal to '0' +``` + +#### 1.2.20 Ensure that the `--secure-port` argument is not set to `0` (Scored) + +**Result:** PASS + +**Remediation:** +Edit the API server pod specification file `/etc/kubernetes/manifests/kube-apiserver.yaml` +on the master node and either remove the `--secure-port` parameter or +set it to a different **(non-zero)** desired port. + +**Audit:** + +``` bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected result**: + +``` bash +6443 is greater than 0 OR '--secure-port' is not present +``` + +#### 1.2.21 Ensure that the `--profiling` argument is set to `false` (Scored) + +**Result:** PASS + +**Remediation:** +Edit the API server pod specification file `/etc/kubernetes/manifests/kube-apiserver.yaml` +on the master node and set the below parameter. + +``` bash +--profiling=false +``` + +**Audit:** + +``` bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected result**: + +``` bash +'false' is equal to 'false' +``` + +#### 1.2.22 Ensure that the `--audit-log-path` argument is set (Scored) + +**Result:** PASS + +**Remediation:** +Edit the API server pod specification file `/etc/kubernetes/manifests/kube-apiserver.yaml` +on the master node and set the `--audit-log-path` parameter to a suitable path and +file where you would like audit logs to be written, for example: + +``` bash +--audit-log-path=/var/log/apiserver/audit.log +``` + +**Audit:** + +``` bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected result**: + +``` bash +'--audit-log-path' is present +``` + +#### 1.2.23 Ensure that the `--audit-log-maxage` argument is set to `30` or as appropriate (Scored) + +**Result:** PASS + +**Remediation:** +Edit the API server pod specification file `/etc/kubernetes/manifests/kube-apiserver.yaml` +on the master node and set the `--audit-log-maxage` parameter to `30` or as an appropriate number of days: + +``` bash +--audit-log-maxage=30 +``` + +**Audit:** + +``` bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected result**: + +``` bash +30 is greater or equal to 30 +``` + +#### 1.2.24 Ensure that the `--audit-log-maxbackup` argument is set to `10` or as appropriate (Scored) + +**Result:** PASS + +**Remediation:** +Edit the API server pod specification file `/etc/kubernetes/manifests/kube-apiserver.yaml` +on the master node and set the `--audit-log-maxbackup` parameter to `10` or to an appropriate +value. + +``` bash +--audit-log-maxbackup=10 +``` + +**Audit:** + +``` bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected result**: + +``` bash +10 is greater or equal to 10 +``` + +#### 1.2.25 Ensure that the `--audit-log-maxsize` argument is set to `100` or as appropriate (Scored) + +**Result:** PASS + +**Remediation:** +Edit the API server pod specification file `/etc/kubernetes/manifests/kube-apiserver.yaml` +on the master node and set the `--audit-log-maxsize` parameter to an appropriate size in **MB**. +For example, to set it as `100` **MB**: + +``` bash +--audit-log-maxsize=100 +``` + +**Audit:** + +``` bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected result**: + +``` bash +100 is greater or equal to 100 +``` + +#### 1.2.26 Ensure that the `--request-timeout` argument is set as appropriate (Scored) + +**Result:** PASS + +**Remediation:** +Edit the API server pod specification file `/etc/kubernetes/manifests/kube-apiserver.yaml` +and set the below parameter as appropriate and if needed. +For example, + +``` bash +--request-timeout=300s +``` + +**Audit:** + +``` bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected result**: + +``` bash +'--request-timeout' is not present OR '--request-timeout' is present +``` + +#### 1.2.27 Ensure that the `--service-account-lookup` argument is set to `true` (Scored) + +**Result:** PASS + +**Remediation:** +Edit the API server pod specification file `/etc/kubernetes/manifests/kube-apiserver.yaml` +on the master node and set the below parameter. + +``` bash +--service-account-lookup=true +``` + +Alternatively, you can delete the `--service-account-lookup` parameter from this file so +that the default takes effect. + +**Audit:** + +``` bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected result**: + +``` bash +'--service-account-lookup' is not present OR 'true' is equal to 'true' +``` + +#### 1.2.28 Ensure that the `--service-account-key-file` argument is set as appropriate (Scored) + +**Result:** PASS + +**Remediation:** +Edit the API server pod specification file `/etc/kubernetes/manifests/kube-apiserver.yaml` +on the master node and set the `--service-account-key-file` parameter +to the public key file for service accounts: + +``` bash +--service-account-key-file= +``` + +**Audit:** + +``` bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected result**: + +``` bash +'--service-account-key-file' is present +``` + +#### 1.2.29 Ensure that the `--etcd-certfile` and `--etcd-keyfile` arguments are set as appropriate (Scored) + +**Result:** PASS + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection between the apiserver and etcd. +Then, edit the API server pod specification file `/etc/kubernetes/manifests/kube-apiserver.yaml` +on the master node and set the **etcd** certificate and **key** file parameters. + +``` bash +--etcd-certfile= +--etcd-keyfile= +``` + +**Audit:** + +``` bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected result**: + +``` bash +'--etcd-certfile' is present AND '--etcd-keyfile' is present +``` + +#### 1.2.30 Ensure that the `--tls-cert-file` and `--tls-private-key-file` arguments are set as appropriate (Scored) + +**Result:** PASS + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection on the apiserver. +Then, edit the API server pod specification file `/etc/kubernetes/manifests/kube-apiserver.yaml` +on the master node and set the TLS certificate and private key file parameters. + +``` bash +--tls-cert-file= +--tls-private-key-file= +``` + +**Audit:** + +``` bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected result**: + +``` bash +'--tls-cert-file' is present AND '--tls-private-key-file' is present +``` + +#### 1.2.31 Ensure that the `--client-ca-file` argument is set as appropriate (Scored) + +**Result:** PASS + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection on the apiserver. +Then, edit the API server pod specification file `/etc/kubernetes/manifests/kube-apiserver.yaml` +on the master node and set the client certificate authority file. + +``` bash +--client-ca-file= +``` + +**Audit:** + +``` bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected result**: + +``` bash +'--client-ca-file' is present +``` + +#### 1.2.32 Ensure that the `--etcd-cafile` argument is set as appropriate (Scored) + +**Result:** PASS + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection between the apiserver and etcd. +Then, edit the API server pod specification file `/etc/kubernetes/manifests/kube-apiserver.yaml` +on the master node and set the etcd certificate authority file parameter. + +``` bash +--etcd-cafile= +``` + +**Audit:** + +``` bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected result**: + +``` bash +'--etcd-cafile' is present +``` + +#### 1.2.33 Ensure that the `--encryption-provider-config` argument is set as appropriate (Scored) + +**Result:** PASS + +**Remediation:** +Follow the Kubernetes documentation and configure a EncryptionConfig file. +Then, edit the API server pod specification file `/etc/kubernetes/manifests/kube-apiserver.yaml` +on the master node and set the `--encryption-provider-config` parameter to the path of that file: + +``` bash +--encryption-provider-config= +``` + +**Audit:** + +``` bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected result**: + +``` bash +'--encryption-provider-config' is present +``` + +#### 1.2.34 Ensure that encryption providers are appropriately configured (Scored) + +**Result:** WARN + +**Remediation:** +Follow the Kubernetes documentation and configure a `EncryptionConfig` file. +In this file, choose **aescbc**, **kms** or **secretbox** as the encryption provider. + +**Audit:** + +``` bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +#### 1.2.35 Ensure that the API Server only makes use of Strong Cryptographic Ciphers (Not Scored) + +**Result:** PASS + +**Remediation:** +Edit the API server pod specification file `/etc/kubernetes/manifests/kube-apiserver.yaml` +on the master node and set the below parameter. + +``` bash +--tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 +``` + +**Audit:** + +``` bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected result**: + +``` bash +'TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256' has 'TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256' +``` + +### 1.3 Controller Manager + +#### 1.3.1 Ensure that the `--terminated-pod-gc-threshold` argument is set as appropriate (Scored) + +**Result:** PASS + +**Remediation:** +Edit the Controller Manager pod specification file `/etc/kubernetes/manifests/kube-controller-manager.yaml` +on the master node and set the `--terminated-pod-gc-threshold` to an appropriate threshold, +for example: + +``` bash +--terminated-pod-gc-threshold=10 +``` + +**Audit:** + +``` bash +/bin/ps -ef | grep kube-controller-manager | grep -v grep +``` + +**Expected result**: + +``` bash +'--terminated-pod-gc-threshold' is present +``` + +#### 1.3.2 Ensure that the `--profiling` argument is set to false (Scored) + +**Result:** PASS + +**Remediation:** +Edit the Controller Manager pod specification file `/etc/kubernetes/manifests/kube-controller-manager.yaml` +on the master node and set the below parameter. + +``` bash +--profiling=false +``` + +**Audit:** + +``` bash +/bin/ps -ef | grep kube-controller-manager | grep -v grep +``` + +**Expected result**: + +``` bash +'false' is equal to 'false' +``` + +#### 1.3.3 Ensure that the `--use-service-account-credentials` argument is set to `true` (Scored) + +**Result:** PASS + +**Remediation:** +Edit the Controller Manager pod specification file `/etc/kubernetes/manifests/kube-controller-manager.yaml` +on the master node to set the below parameter. + +``` bash +--use-service-account-credentials=true +``` + +**Audit:** + +``` bash +/bin/ps -ef | grep kube-controller-manager | grep -v grep +``` + +**Expected result**: + +``` bash +'true' is not equal to 'false' +``` + +#### 1.3.4 Ensure that the `--service-account-private-key-file` argument is set as appropriate (Scored) + +**Result:** PASS + +**Remediation:** +Edit the Controller Manager pod specification file `/etc/kubernetes/manifests/kube-controller-manager.yaml` +on the master node and set the `--service-account-private-key-file` parameter +to the private key file for service accounts. + +``` bash +--service-account-private-key-file= +``` + +**Audit:** + +``` bash +/bin/ps -ef | grep kube-controller-manager | grep -v grep +``` + +**Expected result**: + +``` bash +'--service-account-private-key-file' is present +``` + +#### 1.3.5 Ensure that the `--root-ca-file` argument is set as appropriate (Scored) + +**Result:** PASS + +**Remediation:** +Edit the Controller Manager pod specification file `/etc/kubernetes/manifests/kube-controller-manager.yaml` +on the master node and set the `--root-ca-file` parameter to the certificate bundle file`. + +``` bash +--root-ca-file= +``` + +**Audit:** + +``` bash +/bin/ps -ef | grep kube-controller-manager | grep -v grep +``` + +**Expected result**: + +``` bash +'--root-ca-file' is present +``` + +#### 1.3.6 Ensure that the `RotateKubeletServerCertificate` argument is set to `true` (Scored) + +**Result:** PASS + +**Remediation:** +Edit the Controller Manager pod specification file `/etc/kubernetes/manifests/kube-controller-manager.yaml` +on the master node and set the `--feature-gates` parameter to include `RotateKubeletServerCertificate=true`. + +``` bash +--feature-gates=RotateKubeletServerCertificate=true +``` + +**Audit:** + +``` bash +/bin/ps -ef | grep kube-controller-manager | grep -v grep +``` + +**Expected result**: + +``` bash +'RotateKubeletServerCertificate=true' is equal to 'RotateKubeletServerCertificate=true' +``` + +#### 1.3.7 Ensure that the `--bind-address argument` is set to `127.0.0.1` (Scored) + +**Result:** PASS + +**Remediation:** +Edit the Controller Manager pod specification file `/etc/kubernetes/manifests/kube-controller-manager.yaml` +on the master node and ensure the correct value for the `--bind-address` parameter. + +**Audit:** + +``` bash +/bin/ps -ef | grep kube-controller-manager | grep -v grep +``` + +**Expected result**: + +``` bash +'--bind-address' is present OR '--bind-address' is not present +``` + +### 1.4 Scheduler + +#### 1.4.1 Ensure that the `--profiling` argument is set to `false` (Scored) + +**Result:** PASS + +**Remediation:** +Edit the Scheduler pod specification file `/etc/kubernetes/manifests/kube-scheduler.yaml` file +on the master node and set the below parameter. + +``` bash +--profiling=false +``` + +**Audit:** + +``` bash +/bin/ps -ef | grep kube-scheduler | grep -v grep +``` + +**Expected result**: + +``` bash +'false' is equal to 'false' +``` + +#### 1.4.2 Ensure that the `--bind-address` argument is set to `127.0.0.1` (Scored) + +**Result:** PASS + +**Remediation:** +Edit the Scheduler pod specification file `/etc/kubernetes/manifests/kube-scheduler.yaml` +on the master node and ensure the correct value for the `--bind-address` parameter. + +**Audit:** + +``` bash +/bin/ps -ef | grep kube-scheduler | grep -v grep +``` + +**Expected result**: + +``` bash +'--bind-address' is present OR '--bind-address' is not present +``` + +## 2 Etcd Node Configuration +### 2 Etcd Node Configuration Files + +#### 2.1 Ensure that the `--cert-file` and `--key-file` arguments are set as appropriate (Scored) + +**Result:** PASS + +**Remediation:** +Follow the etcd service documentation and configure TLS encryption. +Then, edit the etcd pod specification file `/etc/kubernetes/manifests/etcd.yaml` +on the master node and set the below parameters. + +``` bash +--cert-file= +--key-file= +``` + +**Audit:** + +``` bash +/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep +``` + +**Expected result**: + +``` bash +'--cert-file' is present AND '--key-file' is present +``` + +#### 2.2 Ensure that the `--client-cert-auth` argument is set to `true` (Scored) + +**Result:** PASS + +**Remediation:** +Edit the etcd pod specification file `/etc/kubernetes/manifests/etcd.yaml` on the master +node and set the below parameter. + +``` bash +--client-cert-auth="true" +``` + +**Audit:** + +``` bash +/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep +``` + +**Expected result**: + +``` bash +'true' is equal to 'true' +``` + +#### 2.3 Ensure that the `--auto-tls` argument is not set to `true` (Scored) + +**Result:** PASS + +**Remediation:** +Edit the etcd pod specification file `/etc/kubernetes/manifests/etcd.yaml` on the master +node and either remove the `--auto-tls` parameter or set it to `false`. + +``` bash + --auto-tls=false +``` + +**Audit:** + +``` bash +/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep +``` + +**Expected result**: + +``` bash +'--auto-tls' is not present OR '--auto-tls' is not present +``` + +#### 2.4 Ensure that the `--peer-cert-file` and `--peer-key-file` arguments are set as appropriate (Scored) + +**Result:** PASS + +**Remediation:** +Follow the etcd service documentation and configure peer TLS encryption as appropriate +for your etcd cluster. Then, edit the etcd pod specification file `/etc/kubernetes/manifests/etcd.yaml` on the +master node and set the below parameters. + +``` bash +--peer-client-file= +--peer-key-file= +``` + +**Audit:** + +``` bash +/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep +``` + +**Expected result**: + +``` bash +'--peer-cert-file' is present AND '--peer-key-file' is present +``` + +#### 2.5 Ensure that the `--peer-client-cert-auth` argument is set to `true` (Scored) + +**Result:** PASS + +**Remediation:** +Edit the etcd pod specification file `/etc/kubernetes/manifests/etcd.yaml` on the master +node and set the below parameter. + +``` bash +--peer-client-cert-auth=true +``` + +**Audit:** + +``` bash +/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep +``` + +**Expected result**: + +``` bash +'true' is equal to 'true' +``` + +#### 2.6 Ensure that the `--peer-auto-tls` argument is not set to `true` (Scored) + +**Result:** PASS + +**Remediation:** +Edit the etcd pod specification file `/etc/kubernetes/manifests/etcd.yaml` on the master +node and either remove the `--peer-auto-tls` parameter or set it to `false`. + +``` bash +--peer-auto-tls=false +``` + +**Audit:** + +``` bash +/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep +``` + +**Expected result**: + +``` bash +'--peer-auto-tls' is not present OR '--peer-auto-tls' is present +``` + +#### 2.7 Ensure that a unique Certificate Authority is used for etcd (Not Scored) + +**Result:** PASS + +**Remediation:** +[Manual test] +Follow the etcd documentation and create a dedicated certificate authority setup for the +etcd service. +Then, edit the etcd pod specification file `/etc/kubernetes/manifests/etcd.yaml` on the +master node and set the below parameter. + +``` bash +--trusted-ca-file= +``` + +**Audit:** + +``` bash +/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep +``` + +**Expected result**: + +``` bash +'--trusted-ca-file' is present +``` + +## 3 Control Plane Configuration +### 3.1 Authentication and Authorization + +#### 3.1.1 Client certificate authentication should not be used for users (Not Scored) + +**Result:** WARN + +**Remediation:** +Alternative mechanisms provided by Kubernetes such as the use of OIDC should be +implemented in place of client certificates. + +### 3.2 Logging + +#### 3.2.1 Ensure that a minimal audit policy is created (Scored) + +**Result:** WARN + +**Remediation:** +Create an audit policy file for your cluster. + +#### 3.2.2 Ensure that the audit policy covers key security concerns (Not Scored) + +**Result:** WARN + +**Remediation:** +Consider modification of the audit policy in use on the cluster to include these items, at a +minimum. + +## 4 Worker Node Security Configuration +### 4.1 Worker Node Configuration Files + +#### 4.1.1 Ensure that the kubelet service file permissions are set to `644` or more restrictive (Scored) + +**Result:** INFO + +**Remediation:** +RKE doesn’t require or maintain a configuration file for the kubelet service. All configuration is passed in as arguments at container run time. + +#### 4.1.2 Ensure that the kubelet service file ownership is set to `root:root` (Scored) + +**Result:** INFO + +**Remediation:** +RKE doesn’t require or maintain a configuration file for the kubelet service. All configuration is passed in as arguments at container run time. + +#### 4.1.3 Ensure that the proxy kubeconfig file permissions are set to `644` or more restrictive (Scored) + +**Result:** INFO + +**Remediation:** +RKE doesn’t require or maintain a configuration file for the proxy service. All configuration is passed in as arguments at container run time. + +#### 4.1.4 Ensure that the proxy kubeconfig file ownership is set to `root:root` (Scored) + +**Result:** INFO + +**Remediation:** +RKE doesn’t require or maintain a configuration file for the proxy service. All configuration is passed in as arguments at container run time. + +#### 4.1.5 Ensure that the kubelet.conf file permissions are set to `644` or more restrictive (Scored) + +**Result:** INFO + +**Remediation:** +RKE doesn’t require or maintain a configuration file for the kubelet service. All configuration is passed in as arguments at container run time. + +#### 4.1.6 Ensure that the kubelet.conf file ownership is set to `root:root` (Scored) + +**Result:** INFO + +**Remediation:** +RKE doesn’t require or maintain a configuration file for the kubelet service. All configuration is passed in as arguments at container run time. + +#### 4.1.7 Ensure that the certificate authorities file permissions are set to `644` or more restrictive (Scored) + +**Result:** WARN + +**Remediation:** +Run the following command to modify the file permissions of the + +``` bash +--client-ca-file chmod 644 +``` + +#### 4.1.8 Ensure that the client certificate authorities file ownership is set to `root:root` (Scored) + +**Result:** PASS + +**Remediation:** +Run the following command to modify the ownership of the `--client-ca-file`. + +``` bash +chown root:root +``` + +**Audit:** + +``` bash +/bin/sh -c 'if test -e /etc/kubernetes/ssl/kube-ca.pem; then stat -c %U:%G /etc/kubernetes/ssl/kube-ca.pem; fi' +``` + +**Expected result**: + +``` bash +'root:root' is equal to 'root:root' +``` + +#### 4.1.9 Ensure that the kubelet configuration file has permissions set to `644` or more restrictive (Scored) + +**Result:** INFO + +**Remediation:** +RKE doesn’t require or maintain a configuration file for the kubelet service. All configuration is passed in as arguments at container run time. + +#### 4.1.10 Ensure that the kubelet configuration file ownership is set to `root:root` (Scored) + +**Result:** INFO + +**Remediation:** +RKE doesn’t require or maintain a configuration file for the kubelet service. All configuration is passed in as arguments at container run time. + +### 4.2 Kubelet + +#### 4.2.1 Ensure that the `--anonymous-auth argument` is set to false (Scored) + +**Result:** PASS + +**Remediation:** +If using a Kubelet config file, edit the file to set authentication: `anonymous`: enabled to +`false`. +If using executable arguments, edit the kubelet service file +`/etc/systemd/system/kubelet.service.d/10-kubeadm.conf` on each worker node and +set the below parameter in `KUBELET_SYSTEM_PODS_ARGS` variable. + +``` bash +--anonymous-auth=false +``` + +Based on your system, restart the kubelet service. For example: + +``` bash +systemctl daemon-reload +systemctl restart kubelet.service +``` + +**Audit:** + +``` bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +``` bash +/bin/cat /var/lib/kubelet/config.yaml +``` + +**Expected result**: + +``` bash +'false' is equal to 'false' +``` + +#### 4.2.2 Ensure that the `--authorization-mode` argument is not set to `AlwaysAllow` (Scored) + +**Result:** PASS + +**Remediation:** +If using a Kubelet config file, edit the file to set authorization: `mode` to `Webhook`. If +using executable arguments, edit the kubelet service file +`/etc/systemd/system/kubelet.service.d/10-kubeadm.conf` on each worker node and +set the below parameter in `KUBELET_AUTHZ_ARGS` variable. + +``` bash +--authorization-mode=Webhook +``` + +Based on your system, restart the kubelet service. For example: + +``` bash +systemctl daemon-reload +systemctl restart kubelet.service +``` + +**Audit:** + +``` bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +``` bash +/bin/cat /var/lib/kubelet/config.yaml +``` + +**Expected result**: + +``` bash +'Webhook' not have 'AlwaysAllow' +``` + +#### 4.2.3 Ensure that the `--client-ca-file` argument is set as appropriate (Scored) + +**Result:** PASS + +**Remediation:** +If using a Kubelet config file, edit the file to set authentication: `x509`: `clientCAFile` to +the location of the client CA file. +If using command line arguments, edit the kubelet service file +`/etc/systemd/system/kubelet.service.d/10-kubeadm.conf` on each worker node and +set the below parameter in `KUBELET_AUTHZ_ARGS` variable. + +``` bash +--client-ca-file= +``` + +Based on your system, restart the kubelet service. For example: + +``` bash +systemctl daemon-reload +systemctl restart kubelet.service +``` + +**Audit:** + +``` bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +``` bash +/bin/cat /var/lib/kubelet/config.yaml +``` + +**Expected result**: + +``` bash +'--client-ca-file' is present +``` + +#### 4.2.4 Ensure that the `--read-only-port` argument is set to `0` (Scored) + +**Result:** PASS + +**Remediation:** +If using a Kubelet config file, edit the file to set `readOnlyPort` to `0`. +If using command line arguments, edit the kubelet service file +`/etc/systemd/system/kubelet.service.d/10-kubeadm.conf` on each worker node and +set the below parameter in `KUBELET_SYSTEM_PODS_ARGS` variable. + +``` bash +--read-only-port=0 +``` + +Based on your system, restart the kubelet service. For example: + +``` bash +systemctl daemon-reload +systemctl restart kubelet.service +``` + +**Audit:** + +``` bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +``` bash +/bin/cat /var/lib/kubelet/config.yaml +``` + +**Expected result**: + +``` bash +'0' is equal to '0' +``` + +#### 4.2.5 Ensure that the `--streaming-connection-idle-timeout` argument is not set to `0` (Scored) + +**Result:** PASS + +**Remediation:** +If using a Kubelet config file, edit the file to set `streamingConnectionIdleTimeout` to a +value other than `0`. +If using command line arguments, edit the kubelet service file +`/etc/systemd/system/kubelet.service.d/10-kubeadm.conf` on each worker node and +set the below parameter in `KUBELET_SYSTEM_PODS_ARGS` variable. + +``` bash +--streaming-connection-idle-timeout=5m +``` + +Based on your system, restart the kubelet service. For example: + +``` bash +systemctl daemon-reload +systemctl restart kubelet.service +``` + +**Audit:** + +``` bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +``` bash +/bin/cat /var/lib/kubelet/config.yaml +``` + +**Expected result**: + +``` bash +'1800s' is not equal to '0' OR '--streaming-connection-idle-timeout' is not present +``` + +#### 4.2.6 Ensure that the ```--protect-kernel-defaults``` argument is set to `true` (Scored) + +**Result:** PASS + +**Remediation:** +If using a Kubelet config file, edit the file to set `protectKernelDefaults`: `true`. +If using command line arguments, edit the kubelet service file +`/etc/systemd/system/kubelet.service.d/10-kubeadm.conf` on each worker node and +set the below parameter in `KUBELET_SYSTEM_PODS_ARGS` variable. + +``` bash +--protect-kernel-defaults=true +``` + +Based on your system, restart the kubelet service. For example: + +``` bash +systemctl daemon-reload +systemctl restart kubelet.service +``` + +**Audit:** + +``` bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +``` bash +/bin/cat /var/lib/kubelet/config.yaml +``` + +**Expected result**: + +``` bash +'true' is equal to 'true' +``` + +#### 4.2.7 Ensure that the `--make-iptables-util-chains` argument is set to `true` (Scored) + +**Result:** PASS + +**Remediation:** +If using a Kubelet config file, edit the file to set `makeIPTablesUtilChains`: `true`. +If using command line arguments, edit the kubelet service file +`/etc/systemd/system/kubelet.service.d/10-kubeadm.conf` on each worker node and +remove the `--make-iptables-util-chains` argument from the +`KUBELET_SYSTEM_PODS_ARGS` variable. +Based on your system, restart the kubelet service. For example: + +```bash +systemctl daemon-reload +systemctl restart kubelet.service +``` + +**Audit:** + +``` bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +``` bash +/bin/cat /var/lib/kubelet/config.yaml +``` + +**Expected result**: + +``` bash +'true' is equal to 'true' OR '--make-iptables-util-chains' is not present +``` + +#### 4.2.8 Ensure that the `--hostname-override` argument is not set (Not Scored) + +**Result:** WARN + +**Remediation:** +Edit the kubelet service file `/etc/systemd/system/kubelet.service.d/10-kubeadm.conf` +on each worker node and remove the `--hostname-override` argument from the +`KUBELET_SYSTEM_PODS_ARGS` variable. +Based on your system, restart the kubelet service. For example: + +``` bash +systemctl daemon-reload +systemctl restart kubelet.service +``` + +**Audit:** + +``` bash +/bin/ps -fC kubelet +``` + +#### 4.2.9 Ensure that the `--event-qps` argument is set to `0` or a level which ensures appropriate event capture (Not Scored) + +**Result:** PASS + +**Remediation:** +If using a Kubelet config file, edit the file to set `eventRecordQPS`: to an appropriate level. +If using command line arguments, edit the kubelet service file +`/etc/systemd/system/kubelet.service.d/10-kubeadm.conf` on each worker node and +set the below parameter in `KUBELET_SYSTEM_PODS_ARGS` variable. +Based on your system, restart the kubelet service. For example: + +``` bash +systemctl daemon-reload +systemctl restart kubelet.service +``` + +**Audit:** + +``` bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +``` bash +/bin/cat /var/lib/kubelet/config.yaml +``` + +**Expected result**: + +``` bash +'0' is equal to '0' +``` + +#### 4.2.10 Ensure that the `--tls-cert-file` and `--tls-private-key-file` arguments are set as appropriate (Scored) + +**Result:** INFO + +**Remediation:** +RKE doesn’t require or maintain a configuration file for the kubelet service. All configuration is passed in as arguments at container run time. + +#### 4.2.11 Ensure that the `--rotate-certificates` argument is not set to `false` (Scored) + +**Result:** PASS + +**Remediation:** +If using a Kubelet config file, edit the file to add the line `rotateCertificates`: `true` or +remove it altogether to use the default value. +If using command line arguments, edit the kubelet service file +`/etc/systemd/system/kubelet.service.d/10-kubeadm.conf` on each worker node and +remove `--rotate-certificates=false` argument from the `KUBELET_CERTIFICATE_ARGS` +variable. +Based on your system, restart the kubelet service. For example: + +``` bash +systemctl daemon-reload +systemctl restart kubelet.service +``` + +**Audit:** + +``` bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +``` bash +/bin/cat /var/lib/kubelet/config.yaml +``` + +**Expected result**: + +``` bash +'--rotate-certificates' is present OR '--rotate-certificates' is not present +``` + +#### 4.2.12 Ensure that the `RotateKubeletServerCertificate` argument is set to `true` (Scored) + +**Result:** PASS + +**Remediation:** +Edit the kubelet service file `/etc/systemd/system/kubelet.service.d/10-kubeadm.conf` +on each worker node and set the below parameter in `KUBELET_CERTIFICATE_ARGS` variable. + +``` bash +--feature-gates=RotateKubeletServerCertificate=true +``` + +Based on your system, restart the kubelet service. For example: + +``` bash +systemctl daemon-reload +systemctl restart kubelet.service +``` + +**Audit:** + +``` bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +``` bash +/bin/cat /var/lib/kubelet/config.yaml +``` + +**Expected result**: + +``` bash +'true' is equal to 'true' +``` + +#### 4.2.13 Ensure that the Kubelet only makes use of Strong Cryptographic Ciphers (Not Scored) + +**Result:** PASS + +**Remediation:** +If using a Kubelet config file, edit the file to set `TLSCipherSuites`: to + +``` bash +TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 +``` + +or to a subset of these values. +If using executable arguments, edit the kubelet service file +`/etc/systemd/system/kubelet.service.d/10-kubeadm.conf` on each worker node and +set the `--tls-cipher-suites` parameter as follows, or to a subset of these values. + +``` bash +--tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 +``` + +Based on your system, restart the kubelet service. For example: + +``` bash +systemctl daemon-reload +systemctl restart kubelet.service +``` + +**Audit:** + +``` bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +``` bash +/bin/cat /var/lib/kubelet/config.yaml +``` + +**Expected result**: + +``` bash +'TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256' contains valid elements from 'TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256' +``` + +## 5 Kubernetes Policies +### 5.1 RBAC and Service Accounts + +#### 5.1.1 Ensure that the cluster-admin role is only used where required (Not Scored) + +**Result:** WARN + +**Remediation:** +Identify all `clusterrolebindings` to the `cluster-admin` role. Check if they are used and +if they need this role or if they could use a role with fewer privileges. +Where possible, first bind users to a lower privileged role and then remove the +`clusterrolebinding` to the `cluster-admin` role : + +``` bash +kubectl delete clusterrolebinding [name] +``` + +#### 5.1.2 Minimize access to secrets (Not Scored) + +**Result:** WARN + +**Remediation:** +Where possible, remove `get`, `list` and `watch` access to secret objects in the cluster. + +#### 5.1.3 Minimize wildcard use in Roles and ClusterRoles (Not Scored) + +**Result:** WARN + +**Remediation:** +Where possible replace any use of wildcards in `clusterroles` and roles with specific +objects or actions. + +#### 5.1.4 Minimize access to create pods (Not Scored) + +**Result:** WARN + +#### 5.1.5 Ensure that default service accounts are not actively used. (Scored) + +**Result:** WARN + +**Remediation:** +Create explicit service accounts wherever a Kubernetes workload requires specific access +to the Kubernetes API server. +Modify the configuration of each default service account to include this value + +``` bash +automountServiceAccountToken: false +``` + +#### 5.1.6 Ensure that Service Account Tokens are only mounted where necessary (Not Scored) + +**Result:** WARN + +**Remediation:** +Modify the definition of pods and service accounts which do not need to mount service +account tokens to disable it. + +### 5.2 Pod Security Policies + +#### 5.2.1 Minimize the admission of privileged containers (Not Scored) + +**Result:** WARN + +**Remediation:** +Create a PSP as described in the Kubernetes documentation, ensuring that +the `.spec.privileged` field is omitted or set to `false`. + +#### 5.2.2 Minimize the admission of containers wishing to share the host process ID namespace (Scored) + +**Result:** WARN + +**Remediation:** +Create a PSP as described in the Kubernetes documentation, ensuring that the +`.spec.hostPID` field is omitted or set to `false`. + +#### 5.2.3 Minimize the admission of containers wishing to share the host IPC namespace (Scored) + +**Result:** WARN + +**Remediation:** +Create a PSP as described in the Kubernetes documentation, ensuring that the +`.spec.hostIPC` field is omitted or set to `false`. + +#### 5.2.4 Minimize the admission of containers wishing to share the host network namespace (Scored) + +**Result:** WARN + +**Remediation:** +Create a PSP as described in the Kubernetes documentation, ensuring that the +`.spec.hostNetwork` field is omitted or set to `false`. + +#### 5.2.5 Minimize the admission of containers with `allowPrivilegeEscalation` (Scored) + +**Result:** WARN + +**Remediation:** +Create a PSP as described in the Kubernetes documentation, ensuring that the +`.spec.allowPrivilegeEscalation` field is omitted or set to `false`. + +#### 5.2.6 Minimize the admission of root containers (Not Scored) + +**Result:** WARN + +**Remediation:** +Create a PSP as described in the Kubernetes documentation, ensuring that the +`.spec.runAsUser.rule` is set to either `MustRunAsNonRoot` or `MustRunAs` with the range of +UIDs not including `0`. + +#### 5.2.7 Minimize the admission of containers with the `NET_RAW` capability (Not Scored) + +**Result:** WARN + +**Remediation:** +Create a PSP as described in the Kubernetes documentation, ensuring that the +`.spec.requiredDropCapabilities` is set to include either `NET_RAW` or `ALL`. + +#### 5.2.8 Minimize the admission of containers with added capabilities (Not Scored) + +**Result:** WARN + +**Remediation:** +Ensure that `allowedCapabilities` is not present in PSPs for the cluster unless +it is set to an empty array. + +#### 5.2.9 Minimize the admission of containers with capabilities assigned (Not Scored) + +**Result:** WARN + +**Remediation:** +Review the use of capabilites in applications runnning on your cluster. Where a namespace +contains applicaions which do not require any Linux capabities to operate consider adding +a PSP which forbids the admission of containers which do not drop all capabilities. + +### 5.3 Network Policies and CNI + +#### 5.3.1 Ensure that the CNI in use supports Network Policies (Not Scored) + +**Result:** WARN + +**Remediation:** +If the CNI plugin in use does not support network policies, consideration should be given to +making use of a different plugin, or finding an alternate mechanism for restricting traffic +in the Kubernetes cluster. + +#### 5.3.2 Ensure that all Namespaces have Network Policies defined (Scored) + +**Result:** WARN + +**Remediation:** +Follow the documentation and create `NetworkPolicy` objects as you need them. + +### 5.4 Secrets Management + +#### 5.4.1 Prefer using secrets as files over secrets as environment variables (Not Scored) + +**Result:** WARN + +**Remediation:** +if possible, rewrite application code to read secrets from mounted secret files, rather than +from environment variables. + +#### 5.4.2 Consider external secret storage (Not Scored) + +**Result:** WARN + +**Remediation:** +Refer to the secrets management options offered by your cloud provider or a third-party +secrets management solution. + +### 5.5 Extensible Admission Control + +#### 5.5.1 Configure Image Provenance using `ImagePolicyWebhook` admission controller (Not Scored) + +**Result:** WARN + +**Remediation:** +Follow the Kubernetes documentation and setup image provenance. + +### 5.6 General Policies + +#### 5.6.1 Create administrative boundaries between resources using namespaces (Not Scored) + +**Result:** WARN + +**Remediation:** +Follow the documentation and create namespaces for objects in your deployment as you need +them. + +#### 5.6.2 Ensure that the seccomp profile is set to docker/default in your pod definitions (Not Scored) + +**Result:** WARN + +**Remediation:** +Seccomp is an alpha feature currently. By default, all alpha features are disabled. So, you +would need to enable alpha features in the apiserver by passing `"--feature- +gates=AllAlpha=true"` argument. +Edit the `/etc/kubernetes/apiserver` file on the master node and set the `KUBE_API_ARGS` +parameter to `"--feature-gates=AllAlpha=true"` +`KUBE_API_ARGS="--feature-gates=AllAlpha=true"` +Based on your system, restart the kube-apiserver service. For example: + +``` bash +systemctl restart kube-apiserver.service +``` + +Use annotations to enable the docker/default seccomp profile in your pod definitions. An +example is as below: + +``` bash +apiVersion: v1 +kind: Pod +metadata: + name: trustworthy-pod + annotations: + seccomp.security.alpha.kubernetes.io/pod: docker/default +spec: + containers: + - name: trustworthy-container + image: sotrustworthy:latest +``` + +#### 5.6.3 Apply Security Context to Your Pods and Containers (Not Scored) + +**Result:** WARN + +**Remediation:** +Follow the Kubernetes documentation and apply security contexts to your pods. For a +suggested list of security contexts, you may refer to the CIS Security Benchmark for Docker +Containers. + +#### 5.6.4 The default namespace should not be used (Scored) + +**Result:** WARN + +**Remediation:** +Ensure that namespaces are created to allow for appropriate segregation of Kubernetes +resources and that all new resources are created in a specific namespace. + diff --git a/content/rancher/v2.x/en/security/hardening-2.3.4/_index.md b/content/rancher/v2.x/en/security/hardening-2.3.4/_index.md new file mode 100644 index 00000000000..352430991e0 --- /dev/null +++ b/content/rancher/v2.x/en/security/hardening-2.3.4/_index.md @@ -0,0 +1,389 @@ +--- +title: Hardening Guide - Rancher v2.3.4. +weight: 100 +--- + +### Hardening Guide for Rancher 2.3.4 with Kubernetes 1.15 + +[Click here to download a PDF version of this document](https://releases.rancher.com/documents/security/2.3.4/Rancher_Hardening_Guide.pdf) + +### Overview + +This document provides prescriptive guidance for hardening a production installation of Rancher v2.3.4 with Kubernetes v1.15. It outlines the configurations required to address Kubernetes benchmark controls from the Center for Information Security (CIS). + +For more detail about evaluating a hardened cluster against the official CIS benchmark, refer to the [CIS Benchmark Rancher Self-Assessment Guide - Rancher v2.3.4]({{< baseurl >}}/rancher/v2.x/en/security/benchmark-2.3.4/). + +### Configure Kernel Runtime Parameters + +The folowing `sysctl` configuration is recommended for all nodes type in the cluster. Set the following parameters in `/etc/sysctl.d/90-kubelet.conf`: + +``` bash +vm.overcommit_memory=1 +vm.panic_on_oom=0 +kernel.panic=10 +kernel.panic_on_oops=1 +kernel.keys.root_maxkeys=1000000 +kernel.keys.root_maxbytes=25000000 +``` + +Run `sysctl -p /etc/sysctl.d/90-kubelet.conf` to enable the settings. + +### Configuration Files and Permissions. + +#### kubelet.conf + +**path**: /etc/sysctl.d/kubelet.conf + +**owner**: root:root + +**permissions:** 0644 + +**contents**: + +``` text +vm.overcommit_memory=1 +kernel.panic=10 +kernel.panic_on_oops=1 +``` + +#### admission.yaml + +**path**: /opt/kubernetes/admission.yaml + +**owner**: root:root + +**permissions**: 0600 + +**content**: + +``` yaml +apiVersion: apiserver.k8s.io/v1alpha1 +kind: AdmissionConfiguration +plugins: +- name: EventRateLimit + path: /opt/kubernetes/event.yaml +``` + +#### event.yaml + +**path**: /opt/kubernetes/event.yaml + +**owner**: root:root + +**permissions**: 0600 + +**content**: + +``` yaml +apiVersion: eventratelimit.admission.k8s.io/v1alpha1 +kind: Configuration +limits: +- type: Server + qps: 5000 + burst: 20000 +``` + +#### encryption.yaml + +**path**: /opt/kubernetes/encryption.yaml + +**owner**: root:root + +**permissions**: 0600 + +**content**: + +``` yaml +apiVersion: apiserver.config.k8s.io/v1 +kind: EncryptionConfiguration +resources: + - resources: + - secrets + providers: + - aescbc: + keys: + - name: key1 + secret: + - identity: {} +``` + + +#### audit.yaml + +**path**: /opt/kubernetes/audit.yaml + +**owner**: root:root + +**permissions**: 0600 + +**content**: + +``` yaml +apiVersion: audit.k8s.io/v1beta1 +kind: Policy +rules: +- level: Metadata + +``` + + +### Minimal RKE `config.yml` configuration + +``` yaml +--- +kubernetes_version: v1.15.6-rancher1-2 +services: +- kube_api: + - extra_args: + - admission-control-config-file: "/opt/kubernetes/admission.yaml" + anonymous-auth: 'false' + audit-log-format: json + audit-log-maxage: '30' + audit-log-maxbackup: '10' + audit-log-maxsize: '100' + audit-log-path: "/var/log/kube-audit/audit-log.json" + audit-policy-file: "/opt/kubernetes/audit.yaml" + enable-admission-plugins: ServiceAccount,NamespaceLifecycle,LimitRanger,PersistentVolumeLabel,DefaultStorageClass,ResourceQuota,DefaultTolerationSeconds,AlwaysPullImages,DenyEscalatingExec,NodeRestriction,EventRateLimit,PodSecurityPolicy + encryption-provider-config: "/opt/kubernetes/encryption.yaml" + profiling: 'false' + service-account-lookup: 'true' + tls-cipher-suites: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 + extra_binds: + - "/var/log/kube-audit:/var/log/kube-audit" + - "/opt/kubernetes:/opt/kubernetes" + pod_security_policy: true + kube_controller: + - extra_args: + - address: 127.0.0.1 + feature-gates: RotateKubeletServerCertificate=true + profiling: 'false' + terminated-pod-gc-threshold: '1000' + kubelet: + - extra_args: + - anonymous-auth: 'false' + event-qps: '0' + feature-gates: RotateKubeletServerCertificate=true + make-iptables-util-chains: 'true' + protect-kernel-defaults: 'true' + streaming-connection-idle-timeout: 1800s + tls-cipher-suites: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 + scheduler: + - extra_args: + - address: 127.0.0.1 + profiling: 'false' +addons: | + --- + apiVersion: v1 + kind: Namespace + metadata: + name: ingress-nginx + --- + apiVersion: rbac.authorization.k8s.io/v1 + kind: Role + metadata: + name: default-psp-role + namespace: ingress-nginx + rules: + - apiGroups: + - extensions + resourceNames: + - default-psp + resources: + - podsecuritypolicies + verbs: + - use + --- + apiVersion: rbac.authorization.k8s.io/v1 + kind: RoleBinding + metadata: + name: default-psp-rolebinding + namespace: ingress-nginx + roleRef: + apiGroup: rbac.authorization.k8s.io + kind: Role + name: default-psp-role + subjects: + - apiGroup: rbac.authorization.k8s.io + kind: Group + name: system:serviceaccounts + - apiGroup: rbac.authorization.k8s.io + kind: Group + name: system:authenticated + --- + apiVersion: v1 + kind: Namespace + metadata: + name: cattle-system + --- + apiVersion: rbac.authorization.k8s.io/v1 + kind: Role + metadata: + name: default-psp-role + namespace: cattle-system + rules: + - apiGroups: + - extensions + resourceNames: + - default-psp + resources: + - podsecuritypolicies + verbs: + - use + --- + apiVersion: rbac.authorization.k8s.io/v1 + kind: RoleBinding + metadata: + name: default-psp-rolebinding + namespace: cattle-system + roleRef: + apiGroup: rbac.authorization.k8s.io + kind: Role + name: default-psp-role + subjects: + - apiGroup: rbac.authorization.k8s.io + kind: Group + name: system:serviceaccounts + - apiGroup: rbac.authorization.k8s.io + kind: Group + name: system:authenticated + --- + apiVersion: extensions/v1beta1 + kind: PodSecurityPolicy + metadata: + name: restricted + spec: + requiredDropCapabilities: + - NET_RAW + privileged: false + allowPrivilegeEscalation: false + defaultAllowPrivilegeEscalation: false + fsGroup: + rule: RunAsAny + runAsUser: + rule: MustRunAsNonRoot + seLinux: + rule: RunAsAny + supplementalGroups: + rule: RunAsAny + volumes: + - emptyDir + - secret + - persistentVolumeClaim + - downwardAPI + - configMap + - projected + --- + apiVersion: rbac.authorization.k8s.io/v1 + kind: ClusterRole + metadata: + name: psp:restricted + rules: + - apiGroups: + - extensions + resourceNames: + - restricted + resources: + - podsecuritypolicies + verbs: + - use + --- + apiVersion: rbac.authorization.k8s.io/v1 + kind: ClusterRoleBinding + metadata: + name: psp:restricted + roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: psp:restricted + subjects: + - apiGroup: rbac.authorization.k8s.io + kind: Group + name: system:serviceaccounts + - apiGroup: rbac.authorization.k8s.io + kind: Group + name: system:authenticated + --- + apiVersion: v1 + kind: ServiceAccount + metadata: + name: tiller + namespace: kube-system + --- + apiVersion: rbac.authorization.k8s.io/v1 + kind: ClusterRoleBinding + metadata: + name: tiller + roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: cluster-admin + subjects: + - kind: ServiceAccount + name: tiller + namespace: kube-system +``` + +### Example Ubuntu cloud-config: + +``` yaml +#cloud-config +packages: + - curl + - jq +runcmd: + - curl https://releases.rancher.com/install-docker/18.09.sh | sh + - usermod -aG docker ubuntu + - sysctl -w vm.overcommit_memory=1 + - sysctl -w kernel.panic=10 + - sysctl -w kernel.panic_on_oops=1 +write_files: + - path: /etc/sysctl.d/kubelet.conf + owner: root:root + permissions: "0644" + content: | + vm.overcommit_memory=1 + kernel.panic=10 + kernel.panic_on_oops=1 + - path: /opt/kubernetes/admission.yaml + owner: root:root + permissions: "0600" + content: | + apiVersion: apiserver.k8s.io/v1alpha1 + kind: AdmissionConfiguration + plugins: + - name: EventRateLimit + path: /opt/kubernetes/event.yaml + - path: /opt/kubernetes/event.yaml + owner: root:root + permissions: "0600" + content: | + apiVersion: eventratelimit.admission.k8s.io/v1alpha1 + kind: Configuration + limits: + - type: Server + qps: 5000 + burst: 20000 + - path: /opt/kubernetes/encryption.yaml + owner: root:root + permissions: "0600" + content: | + apiVersion: apiserver.config.k8s.io/v1 + kind: EncryptionConfiguration + resources: + - resources: + - secrets + providers: + - aescbc: + keys: + - name: key1 + secret: + - identity: {} + - path: /opt/kubernetes/audit.yaml + owner: root:root + permissions: "0600" + content: | + apiVersion: audit.k8s.io/v1beta1 + kind: Policy + rules: + - level: Metadata +``` From 5389832fd33c6eec338e51a9e19a2039538d4e13 Mon Sep 17 00:00:00 2001 From: Nelson Roberts Date: Mon, 3 Feb 2020 16:10:38 -0700 Subject: [PATCH 02/14] EIO-247 final updates for hardening-2.3.4 --- .../en/security/hardening-2.3.4/_index.md | 500 +++++++++++------- 1 file changed, 301 insertions(+), 199 deletions(-) diff --git a/content/rancher/v2.x/en/security/hardening-2.3.4/_index.md b/content/rancher/v2.x/en/security/hardening-2.3.4/_index.md index 352430991e0..6b437ce37f3 100644 --- a/content/rancher/v2.x/en/security/hardening-2.3.4/_index.md +++ b/content/rancher/v2.x/en/security/hardening-2.3.4/_index.md @@ -1,9 +1,18 @@ --- -title: Hardening Guide - Rancher v2.3.4. +title: Hardening Guide v2.3.4 weight: 100 --- -### Hardening Guide for Rancher 2.3.4 with Kubernetes 1.15 +This document provides prescriptive guidance for hardening a production installation of Rancher v2.3.4. It outlines the configurations and controls required to address Kubernetes benchmark controls from the Center for Information Security (CIS). + +> This hardening guide describes how to secure the nodes in your cluster, and it is recommended to follow this guide before installing Kubernetes. + +This hardening guide is intended to be used with specific versions of the CIS Kubernetes Benchmark, Kubernetes, and Rancher: + +Hardening Guide Version | Rancher Version | CIS Benchmark Version | Kubernetes Version +------------------------|----------------|-----------------------|------------------ +Hardening Guide v2.3.4 | Rancher v2.3.4 | Benchmark v1.5 | Kubernetes 1.15 + [Click here to download a PDF version of this document](https://releases.rancher.com/documents/security/2.3.4/Rancher_Hardening_Guide.pdf) @@ -126,216 +135,309 @@ rules: ``` +### Configure `etcd` user and `data-dir` permissions -### Minimal RKE `config.yml` configuration +#### create `etcd` user and `group` -``` yaml ---- -kubernetes_version: v1.15.6-rancher1-2 -services: -- kube_api: - - extra_args: - - admission-control-config-file: "/opt/kubernetes/admission.yaml" - anonymous-auth: 'false' - audit-log-format: json - audit-log-maxage: '30' - audit-log-maxbackup: '10' - audit-log-maxsize: '100' - audit-log-path: "/var/log/kube-audit/audit-log.json" - audit-policy-file: "/opt/kubernetes/audit.yaml" - enable-admission-plugins: ServiceAccount,NamespaceLifecycle,LimitRanger,PersistentVolumeLabel,DefaultStorageClass,ResourceQuota,DefaultTolerationSeconds,AlwaysPullImages,DenyEscalatingExec,NodeRestriction,EventRateLimit,PodSecurityPolicy - encryption-provider-config: "/opt/kubernetes/encryption.yaml" - profiling: 'false' - service-account-lookup: 'true' - tls-cipher-suites: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 - extra_binds: - - "/var/log/kube-audit:/var/log/kube-audit" - - "/opt/kubernetes:/opt/kubernetes" - pod_security_policy: true - kube_controller: - - extra_args: - - address: 127.0.0.1 - feature-gates: RotateKubeletServerCertificate=true - profiling: 'false' - terminated-pod-gc-threshold: '1000' - kubelet: - - extra_args: - - anonymous-auth: 'false' - event-qps: '0' - feature-gates: RotateKubeletServerCertificate=true - make-iptables-util-chains: 'true' - protect-kernel-defaults: 'true' - streaming-connection-idle-timeout: 1800s - tls-cipher-suites: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 - scheduler: - - extra_args: - - address: 127.0.0.1 - profiling: 'false' -addons: | - --- - apiVersion: v1 - kind: Namespace - metadata: - name: ingress-nginx - --- - apiVersion: rbac.authorization.k8s.io/v1 - kind: Role - metadata: - name: default-psp-role - namespace: ingress-nginx - rules: - - apiGroups: - - extensions - resourceNames: - - default-psp - resources: - - podsecuritypolicies - verbs: - - use - --- - apiVersion: rbac.authorization.k8s.io/v1 - kind: RoleBinding - metadata: - name: default-psp-rolebinding - namespace: ingress-nginx - roleRef: - apiGroup: rbac.authorization.k8s.io - kind: Role - name: default-psp-role - subjects: - - apiGroup: rbac.authorization.k8s.io - kind: Group - name: system:serviceaccounts - - apiGroup: rbac.authorization.k8s.io - kind: Group - name: system:authenticated - --- - apiVersion: v1 - kind: Namespace - metadata: - name: cattle-system - --- - apiVersion: rbac.authorization.k8s.io/v1 - kind: Role - metadata: - name: default-psp-role - namespace: cattle-system - rules: - - apiGroups: - - extensions - resourceNames: - - default-psp - resources: - - podsecuritypolicies - verbs: - - use - --- - apiVersion: rbac.authorization.k8s.io/v1 - kind: RoleBinding - metadata: - name: default-psp-rolebinding - namespace: cattle-system - roleRef: - apiGroup: rbac.authorization.k8s.io - kind: Role - name: default-psp-role - subjects: - - apiGroup: rbac.authorization.k8s.io - kind: Group - name: system:serviceaccounts - - apiGroup: rbac.authorization.k8s.io - kind: Group - name: system:authenticated - --- - apiVersion: extensions/v1beta1 - kind: PodSecurityPolicy - metadata: - name: restricted - spec: - requiredDropCapabilities: - - NET_RAW - privileged: false - allowPrivilegeEscalation: false - defaultAllowPrivilegeEscalation: false - fsGroup: - rule: RunAsAny - runAsUser: - rule: MustRunAsNonRoot - seLinux: - rule: RunAsAny - supplementalGroups: - rule: RunAsAny - volumes: - - emptyDir - - secret - - persistentVolumeClaim - - downwardAPI - - configMap - - projected - --- - apiVersion: rbac.authorization.k8s.io/v1 - kind: ClusterRole - metadata: - name: psp:restricted - rules: - - apiGroups: - - extensions - resourceNames: - - restricted - resources: - - podsecuritypolicies - verbs: - - use - --- - apiVersion: rbac.authorization.k8s.io/v1 - kind: ClusterRoleBinding - metadata: - name: psp:restricted - roleRef: - apiGroup: rbac.authorization.k8s.io - kind: ClusterRole - name: psp:restricted - subjects: - - apiGroup: rbac.authorization.k8s.io - kind: Group - name: system:serviceaccounts - - apiGroup: rbac.authorization.k8s.io - kind: Group - name: system:authenticated - --- - apiVersion: v1 - kind: ServiceAccount - metadata: - name: tiller - namespace: kube-system - --- - apiVersion: rbac.authorization.k8s.io/v1 - kind: ClusterRoleBinding - metadata: - name: tiller - roleRef: - apiGroup: rbac.authorization.k8s.io - kind: ClusterRole - name: cluster-admin - subjects: - - kind: ServiceAccount - name: tiller - namespace: kube-system +``` +addgroup --gid 52034 etcd +useradd --comment "etcd service account" --uid 52034 --gid 52034 etcd ``` -### Example Ubuntu cloud-config: +#### create `data-dir` and set permissions +``` +mkdir -p /var/lib/etcd && chown etcd.etcd /var/lib/etcd && chmod 0700 /var/lib/etcd +``` + + +### Hardened RKE `config.yml` configuration + +``` yaml +# +# Cluster Config +# +docker_root_dir: /var/lib/docker +enable_cluster_alerting: false +enable_cluster_monitoring: false +enable_network_policy: false +# +# Rancher Config +# +rancher_kubernetes_engine_config: + addon_job_timeout: 30 + addons: |- + --- + apiVersion: v1 + kind: Namespace + metadata: + name: ingress-nginx + --- + apiVersion: rbac.authorization.k8s.io/v1 + kind: Role + metadata: + name: default-psp-role + namespace: ingress-nginx + rules: + - apiGroups: + - extensions + resourceNames: + - default-psp + resources: + - podsecuritypolicies + verbs: + - use + --- + apiVersion: rbac.authorization.k8s.io/v1 + kind: RoleBinding + metadata: + name: default-psp-rolebinding + namespace: ingress-nginx + roleRef: + apiGroup: rbac.authorization.k8s.io + kind: Role + name: default-psp-role + subjects: + - apiGroup: rbac.authorization.k8s.io + kind: Group + name: system:serviceaccounts + - apiGroup: rbac.authorization.k8s.io + kind: Group + name: system:authenticated + --- + apiVersion: v1 + kind: Namespace + metadata: + name: cattle-system + --- + apiVersion: rbac.authorization.k8s.io/v1 + kind: Role + metadata: + name: default-psp-role + namespace: cattle-system + rules: + - apiGroups: + - extensions + resourceNames: + - default-psp + resources: + - podsecuritypolicies + verbs: + - use + --- + apiVersion: rbac.authorization.k8s.io/v1 + kind: RoleBinding + metadata: + name: default-psp-rolebinding + namespace: cattle-system + roleRef: + apiGroup: rbac.authorization.k8s.io + kind: Role + name: default-psp-role + subjects: + - apiGroup: rbac.authorization.k8s.io + kind: Group + name: system:serviceaccounts + - apiGroup: rbac.authorization.k8s.io + kind: Group + name: system:authenticated + --- + apiVersion: extensions/v1beta1 + kind: PodSecurityPolicy + metadata: + name: restricted + spec: + requiredDropCapabilities: + - NET_RAW + privileged: false + allowPrivilegeEscalation: false + defaultAllowPrivilegeEscalation: false + fsGroup: + rule: RunAsAny + runAsUser: + rule: MustRunAsNonRoot + seLinux: + rule: RunAsAny + supplementalGroups: + rule: RunAsAny + volumes: + - emptyDir + - secret + - persistentVolumeClaim + - downwardAPI + - configMap + - projected + --- + apiVersion: rbac.authorization.k8s.io/v1 + kind: ClusterRole + metadata: + name: psp:restricted + rules: + - apiGroups: + - extensions + resourceNames: + - restricted + resources: + - podsecuritypolicies + verbs: + - use + --- + apiVersion: rbac.authorization.k8s.io/v1 + kind: ClusterRoleBinding + metadata: + name: psp:restricted + roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: psp:restricted + subjects: + - apiGroup: rbac.authorization.k8s.io + kind: Group + name: system:serviceaccounts + - apiGroup: rbac.authorization.k8s.io + kind: Group + name: system:authenticated + --- + apiVersion: v1 + kind: ServiceAccount + metadata: + name: tiller + namespace: kube-system + --- + apiVersion: rbac.authorization.k8s.io/v1 + kind: ClusterRoleBinding + metadata: + name: tiller + roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: cluster-admin + subjects: + - kind: ServiceAccount + name: tiller + namespace: kube-system + ignore_docker_version: true + kubernetes_version: v1.15.6-rancher1-2 +# +# If you are using calico on AWS +# +# network: +# plugin: calico +# calico_network_provider: +# cloud_provider: aws +# +# # To specify flannel interface +# +# network: +# plugin: flannel +# flannel_network_provider: +# iface: eth1 +# +# # To specify flannel interface for canal plugin +# +# network: +# plugin: canal +# canal_network_provider: +# iface: eth1 +# + network: + mtu: 0 + plugin: canal +# +# services: +# kube-api: +# service_cluster_ip_range: 10.43.0.0/16 +# kube-controller: +# cluster_cidr: 10.42.0.0/16 +# service_cluster_ip_range: 10.43.0.0/16 +# kubelet: +# cluster_domain: cluster.local +# cluster_dns_server: 10.43.0.10 +# + services: + etcd: + backup_config: + enabled: false + interval_hours: 12 + retention: 6 + safe_timestamp: false + creation: 12h + extra_args: + data-dir: /var/lib/etcd + extra_binds: + - '/var/lib/etcd:/var/lib/etcd' + gid: 52034 + retention: 72h + snapshot: false + uid: 52034 + kube_api: + always_pull_images: false + extra_args: + admission-control-config-file: /opt/kubernetes/admission.yaml + anonymous-auth: 'false' + audit-log-format: json + audit-log-maxage: '30' + audit-log-maxbackup: '10' + audit-log-maxsize: '100' + audit-log-path: /var/log/kube-audit/audit-log.json + audit-policy-file: /opt/kubernetes/audit.yaml + enable-admission-plugins: >- + ServiceAccount,NamespaceLifecycle,LimitRanger,PersistentVolumeLabel,DefaultStorageClass,ResourceQuota,DefaultTolerationSeconds,AlwaysPullImages,DenyEscalatingExec,NodeRestriction,EventRateLimit,PodSecurityPolicy + encryption-provider-config: /opt/kubernetes/encryption.yaml + profiling: 'false' + service-account-lookup: 'true' + tls-cipher-suites: >- + TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 + extra_binds: + - '/var/log/kube-audit:/var/log/kube-audit' + - '/opt/kubernetes:/opt/kubernetes' + pod_security_policy: true + service_node_port_range: 30000-32767 + kube_controller: + extra_args: + address: 127.0.0.1 + feature-gates: RotateKubeletServerCertificate=true + profiling: 'false' + terminated-pod-gc-threshold: '1000' + kubelet: + extra_args: + anonymous-auth: 'false' + event-qps: '0' + feature-gates: RotateKubeletServerCertificate=true + make-iptables-util-chains: 'true' + protect-kernel-defaults: 'true' + streaming-connection-idle-timeout: 1800s + tls-cipher-suites: >- + TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 + fail_swap_on: false + generate_serving_certificate: true + scheduler: + extra_args: + address: 127.0.0.1 + profiling: 'false' + ssh_agent_auth: false +windows_prefered_cluster: false +``` + +### Hardened Example Ubuntu cloud-config: ``` yaml #cloud-config +package_update: false packages: - curl - jq runcmd: - - curl https://releases.rancher.com/install-docker/18.09.sh | sh - - usermod -aG docker ubuntu - sysctl -w vm.overcommit_memory=1 - sysctl -w kernel.panic=10 - sysctl -w kernel.panic_on_oops=1 + - curl https://releases.rancher.com/install-docker/18.09.sh | sh + - usermod -aG docker ubuntu + - return=1; while [ $return != 0 ]; do sleep 2; docker ps; return=$?; done + - addgroup --gid 52034 etcd + - useradd --comment "etcd service account" --uid 52034 --gid 52034 etcd + - mkdir -p /var/lib/etcd && chown etcd.etcd /var/lib/etcd && chmod 0700 /var/lib/etcd + - ${agent_cmd} --etcd --controlplane --worker + - mkdir /mnt/kube-bench write_files: - path: /etc/sysctl.d/kubelet.conf owner: root:root @@ -376,7 +478,7 @@ write_files: - aescbc: keys: - name: key1 - secret: + secret: LF7YiCFyWqAa2MovOgp42rArBdLBGWdjJpX2knvYAkc= - identity: {} - path: /opt/kubernetes/audit.yaml owner: root:root From 640f3ba69aac26dde1d3ec7fe5e475da5d4a94a1 Mon Sep 17 00:00:00 2001 From: Nelson Roberts Date: Mon, 3 Feb 2020 16:11:21 -0700 Subject: [PATCH 03/14] EIO-247 final updates for benchmark-2.3.4 --- .../en/security/benchmark-2.3.4/_index.md | 326 +++++++++--------- 1 file changed, 170 insertions(+), 156 deletions(-) diff --git a/content/rancher/v2.x/en/security/benchmark-2.3.4/_index.md b/content/rancher/v2.x/en/security/benchmark-2.3.4/_index.md index 669565d9a82..57e25c15e7c 100644 --- a/content/rancher/v2.x/en/security/benchmark-2.3.4/_index.md +++ b/content/rancher/v2.x/en/security/benchmark-2.3.4/_index.md @@ -1,6 +1,6 @@ --- title: CIS Benchmark Rancher Self-Assessment Guide - Rancher v2.3.4 -weight: 103 +weight: 105 --- ### CIS Kubernetes Benchmark 1.5 - Rancher 2.3.4 with Kubernetes 1.15 @@ -9,10 +9,14 @@ weight: 103 #### Overview -The following document scores a Kubernetes 1.15.x RKE cluster provisioned according to the Rancher v2.3.4 hardening guide against the CIS 1.5 Kubernetes benchmark. - This document is a companion to the Rancher v2.3.4 security hardening guide. The hardening guide provides prescriptive guidance for hardening a production installation of Rancher, and this benchmark guide is meant to help you evaluate the level of security of the hardened cluster against each control in the benchmark. +This guide corresponds to specific versions of the hardening guide, Rancher, Kubernetes, and the CIS Benchmark: + +Self Assessment Guide Version | Rancher Version | Hardening Guide Version | Kubernetes Version | CIS Benchmark Version +---------------------------|----------|---------|-------|----- +Self Assessment Guide v2.3.4 | Rancher v2.3.4 | Hardening Guide v2.3.4 | Kubernetes v1.15 | Benchmark v1.5 + Because Rancher and RKE install Kubernetes services as Docker containers, many of the control verification checks in the CIS Kubernetes Benchmark don't apply. This guide will walk through the various controls and provide updated example commands to audit compliance in Rancher-created clusters. This document is to be used by Rancher operators, security teams, auditors and decision makers. @@ -27,14 +31,6 @@ Scoring the commands is different in Rancher Labs than in the CIS Benchmark. Whe When performing the tests, you will need access to the Docker command line on the hosts of all three RKE roles. The commands also make use of the the `jq` command to provide human-readable formatting. -#### Known Scored Control Failures - -The following scored controls do not currently pass, and Rancher Labs is working towards addressing these through future enhancements to the product. - -- 1.1.11 Ensure that the etcd data directory permissions are set to 700 or more restrictive (Scored) -- 1.1.12 Ensure that the etcd data directory ownership is set to etcd:etcd (Scored) -- 1.2.6 Ensure that the --kubelet-certificate-authority argument is set as appropriate (Scored) - ### Controls --- @@ -111,7 +107,7 @@ chmod 644 **Audit:** -``` bash +``` stat -c %a ``` @@ -129,13 +125,13 @@ chown root:root **Audit:** -``` bash +``` stat -c %U:%G ``` #### 1.1.11 Ensure that the etcd data directory permissions are set to `700` or more restrictive (Scored) -**Result:** FAIL +**Result:** PASS **Remediation:** On the etcd server node, get the etcd data directory, passed as an argument `--data-dir`, @@ -153,13 +149,19 @@ chmod 700 /var/lib/etcd **Audit:** -``` bash +``` ps -ef | grep etcd | grep -- --data-dir | sed 's%.*data-dir[= ]\([^ ]*\).*%\1%' | xargs stat -c %a ``` +**Expected result**: + +``` +'700' is equal to '700' +``` + #### 1.1.12 Ensure that the etcd data directory ownership is set to `etcd:etcd` (Scored) -**Result:** FAIL +**Result:** PASS **Remediation:** On the etcd server node, get the etcd data directory, passed as an argument `--data-dir`, @@ -177,10 +179,16 @@ chown etcd:etcd /var/lib/etcd **Audit:** -``` bash +``` ps -ef | grep etcd | grep -- --data-dir | sed 's%.*data-dir[= ]\([^ ]*\).*%\1%' | xargs stat -c %U:%G ``` +**Expected result**: + +``` +'etcd:etcd' is present +``` + #### 1.1.13 Ensure that the `admin.conf` file permissions are set to `644` or more restrictive (Scored) **Result:** INFO @@ -239,7 +247,7 @@ chown -R root:root /etc/kubernetes/pki/ **Audit:** -``` bash +``` ls -laR /etc/kubernetes/pki/ ``` @@ -257,7 +265,7 @@ chmod -R 644 /etc/kubernetes/pki/*.crt **Audit:** -``` bash +``` stat -c %n %a /etc/kubernetes/pki/*.crt ``` @@ -275,7 +283,7 @@ chmod -R 600 /etc/kubernetes/pki/*.key **Audit:** -``` bash +``` stat -c %n %a /etc/kubernetes/pki/*.key ``` @@ -295,13 +303,13 @@ on the master node and set the below parameter. **Audit:** -``` bash +``` /bin/ps -ef | grep kube-apiserver | grep -v grep ``` **Expected result**: -``` bash +``` 'false' is equal to 'false' ``` @@ -316,13 +324,13 @@ on the master node and remove the `--basic-auth-file=` parameter. **Audit:** -``` bash +``` /bin/ps -ef | grep kube-apiserver | grep -v grep ``` **Expected result**: -``` bash +``` '--basic-auth-file' is not present ``` @@ -337,13 +345,13 @@ on the master node and remove the `--token-auth-file=` parameter. **Audit:** -``` bash +``` /bin/ps -ef | grep kube-apiserver | grep -v grep ``` **Expected result**: -``` bash +``` '--token-auth-file' is not present ``` @@ -357,13 +365,13 @@ on the master node and remove the `--kubelet-https` parameter. **Audit:** -``` bash +``` /bin/ps -ef | grep kube-apiserver | grep -v grep ``` **Expected result**: -``` bash +``` '--kubelet-https' is present OR '--kubelet-https' is not present ``` @@ -384,19 +392,19 @@ kubelet client certificate and key parameters as below. **Audit:** -``` bash +``` /bin/ps -ef | grep kube-apiserver | grep -v grep ``` **Expected result**: -``` bash +``` '--kubelet-client-certificate' is present AND '--kubelet-client-key' is present ``` #### 1.2.6 Ensure that the `--kubelet-certificate-authority` argument is set as appropriate (Scored) -**Result:** FAIL +**Result:** PASS **Remediation:** Follow the Kubernetes documentation and setup the TLS connection between @@ -407,10 +415,16 @@ the apiserver and kubelets. Then, edit the API server pod specification file **Audit:** -``` bash +``` /bin/ps -ef | grep kube-apiserver | grep -v grep ``` +**Expected result**: + +``` +'--kubelet-certificate-authority' is present +``` + #### 1.2.7 Ensure that the `--authorization-mode` argument is not set to `AlwaysAllow` (Scored) **Result:** PASS @@ -426,13 +440,13 @@ One such example could be as below. **Audit:** -``` bash +``` /bin/ps -ef | grep kube-apiserver | grep -v grep ``` **Expected result**: -``` bash +``` 'Node,RBAC' not have 'AlwaysAllow' ``` @@ -450,13 +464,13 @@ on the master node and set the `--authorization-mode` parameter to a value that **Audit:** -``` bash +``` /bin/ps -ef | grep kube-apiserver | grep -v grep ``` **Expected result**: -``` bash +``` 'Node,RBAC' has 'Node' ``` @@ -475,13 +489,13 @@ for example: **Audit:** -``` bash +``` /bin/ps -ef | grep kube-apiserver | grep -v grep ``` **Expected result**: -``` bash +``` 'Node,RBAC' has 'RBAC' ``` @@ -501,13 +515,13 @@ and set the below parameters. **Audit:** -``` bash +``` /bin/ps -ef | grep kube-apiserver | grep -v grep ``` **Expected result**: -``` bash +``` 'ServiceAccount,NamespaceLifecycle,LimitRanger,PersistentVolumeLabel,DefaultStorageClass,ResourceQuota,DefaultTolerationSeconds,AlwaysPullImages,DenyEscalatingExec,NodeRestriction,EventRateLimit,PodSecurityPolicy' has 'EventRateLimit' ``` @@ -522,13 +536,13 @@ value that does not include `AlwaysAdmit`. **Audit:** -``` bash +``` /bin/ps -ef | grep kube-apiserver | grep -v grep ``` **Expected result**: -``` bash +``` 'ServiceAccount,NamespaceLifecycle,LimitRanger,PersistentVolumeLabel,DefaultStorageClass,ResourceQuota,DefaultTolerationSeconds,AlwaysPullImages,DenyEscalatingExec,NodeRestriction,EventRateLimit,PodSecurityPolicy' not have 'AlwaysAdmit' OR '--enable-admission-plugins' is not present ``` @@ -547,13 +561,13 @@ on the master node and set the `--enable-admission-plugins` parameter to include **Audit:** -``` bash +``` /bin/ps -ef | grep kube-apiserver | grep -v grep ``` **Expected result**: -``` bash +``` 'ServiceAccount,NamespaceLifecycle,LimitRanger,PersistentVolumeLabel,DefaultStorageClass,ResourceQuota,DefaultTolerationSeconds,AlwaysPullImages,DenyEscalatingExec,NodeRestriction,EventRateLimit,PodSecurityPolicy' has 'AlwaysPullImages' ``` @@ -572,7 +586,7 @@ on the master node and set the `--enable-admission-plugins` parameter to include **Audit:** -``` bash +``` /bin/ps -ef | grep kube-apiserver | grep -v grep ``` @@ -588,13 +602,13 @@ value that does not include `ServiceAccount`. **Audit:** -``` bash +``` /bin/ps -ef | grep kube-apiserver | grep -v grep ``` **Expected result**: -``` bash +``` 'ServiceAccount,NamespaceLifecycle,LimitRanger,PersistentVolumeLabel,DefaultStorageClass,ResourceQuota,DefaultTolerationSeconds,AlwaysPullImages,DenyEscalatingExec,NodeRestriction,EventRateLimit,PodSecurityPolicy' has 'ServiceAccount' OR '--enable-admission-plugins' is not present ``` @@ -609,13 +623,13 @@ ensure it does not include `NamespaceLifecycle`. **Audit:** -``` bash +``` /bin/ps -ef | grep kube-apiserver | grep -v grep ``` **Expected result**: -``` bash +``` '--disable-admission-plugins' is present OR '--disable-admission-plugins' is not present ``` @@ -637,13 +651,13 @@ Then restart the API Server. **Audit:** -``` bash +``` /bin/ps -ef | grep kube-apiserver | grep -v grep ``` **Expected result**: -``` bash +``` 'ServiceAccount,NamespaceLifecycle,LimitRanger,PersistentVolumeLabel,DefaultStorageClass,ResourceQuota,DefaultTolerationSeconds,AlwaysPullImages,DenyEscalatingExec,NodeRestriction,EventRateLimit,PodSecurityPolicy' has 'PodSecurityPolicy' ``` @@ -663,13 +677,13 @@ value that includes `NodeRestriction`. **Audit:** -``` bash +``` /bin/ps -ef | grep kube-apiserver | grep -v grep ``` **Expected result**: -``` bash +``` 'ServiceAccount,NamespaceLifecycle,LimitRanger,PersistentVolumeLabel,DefaultStorageClass,ResourceQuota,DefaultTolerationSeconds,AlwaysPullImages,DenyEscalatingExec,NodeRestriction,EventRateLimit,PodSecurityPolicy' has 'NodeRestriction' ``` @@ -683,13 +697,13 @@ on the master node and remove the `--insecure-bind-address` parameter. **Audit:** -``` bash +``` /bin/ps -ef | grep kube-apiserver | grep -v grep ``` **Expected result**: -``` bash +``` '--insecure-bind-address' is not present ``` @@ -707,13 +721,13 @@ on the master node and set the below parameter. **Audit:** -``` bash +``` /bin/ps -ef | grep kube-apiserver | grep -v grep ``` **Expected result**: -``` bash +``` '0' is equal to '0' ``` @@ -728,13 +742,13 @@ set it to a different **(non-zero)** desired port. **Audit:** -``` bash +``` /bin/ps -ef | grep kube-apiserver | grep -v grep ``` **Expected result**: -``` bash +``` 6443 is greater than 0 OR '--secure-port' is not present ``` @@ -752,13 +766,13 @@ on the master node and set the below parameter. **Audit:** -``` bash +``` /bin/ps -ef | grep kube-apiserver | grep -v grep ``` **Expected result**: -``` bash +``` 'false' is equal to 'false' ``` @@ -777,13 +791,13 @@ file where you would like audit logs to be written, for example: **Audit:** -``` bash +``` /bin/ps -ef | grep kube-apiserver | grep -v grep ``` **Expected result**: -``` bash +``` '--audit-log-path' is present ``` @@ -801,13 +815,13 @@ on the master node and set the `--audit-log-maxage` parameter to `30` or as an a **Audit:** -``` bash +``` /bin/ps -ef | grep kube-apiserver | grep -v grep ``` **Expected result**: -``` bash +``` 30 is greater or equal to 30 ``` @@ -826,13 +840,13 @@ value. **Audit:** -``` bash +``` /bin/ps -ef | grep kube-apiserver | grep -v grep ``` **Expected result**: -``` bash +``` 10 is greater or equal to 10 ``` @@ -851,13 +865,13 @@ For example, to set it as `100` **MB**: **Audit:** -``` bash +``` /bin/ps -ef | grep kube-apiserver | grep -v grep ``` **Expected result**: -``` bash +``` 100 is greater or equal to 100 ``` @@ -876,13 +890,13 @@ For example, **Audit:** -``` bash +``` /bin/ps -ef | grep kube-apiserver | grep -v grep ``` **Expected result**: -``` bash +``` '--request-timeout' is not present OR '--request-timeout' is present ``` @@ -903,13 +917,13 @@ that the default takes effect. **Audit:** -``` bash +``` /bin/ps -ef | grep kube-apiserver | grep -v grep ``` **Expected result**: -``` bash +``` '--service-account-lookup' is not present OR 'true' is equal to 'true' ``` @@ -928,13 +942,13 @@ to the public key file for service accounts: **Audit:** -``` bash +``` /bin/ps -ef | grep kube-apiserver | grep -v grep ``` **Expected result**: -``` bash +``` '--service-account-key-file' is present ``` @@ -954,13 +968,13 @@ on the master node and set the **etcd** certificate and **key** file parameters. **Audit:** -``` bash +``` /bin/ps -ef | grep kube-apiserver | grep -v grep ``` **Expected result**: -``` bash +``` '--etcd-certfile' is present AND '--etcd-keyfile' is present ``` @@ -980,13 +994,13 @@ on the master node and set the TLS certificate and private key file parameters. **Audit:** -``` bash +``` /bin/ps -ef | grep kube-apiserver | grep -v grep ``` **Expected result**: -``` bash +``` '--tls-cert-file' is present AND '--tls-private-key-file' is present ``` @@ -1005,13 +1019,13 @@ on the master node and set the client certificate authority file. **Audit:** -``` bash +``` /bin/ps -ef | grep kube-apiserver | grep -v grep ``` **Expected result**: -``` bash +``` '--client-ca-file' is present ``` @@ -1030,13 +1044,13 @@ on the master node and set the etcd certificate authority file parameter. **Audit:** -``` bash +``` /bin/ps -ef | grep kube-apiserver | grep -v grep ``` **Expected result**: -``` bash +``` '--etcd-cafile' is present ``` @@ -1055,13 +1069,13 @@ on the master node and set the `--encryption-provider-config` parameter to the p **Audit:** -``` bash +``` /bin/ps -ef | grep kube-apiserver | grep -v grep ``` **Expected result**: -``` bash +``` '--encryption-provider-config' is present ``` @@ -1075,7 +1089,7 @@ In this file, choose **aescbc**, **kms** or **secretbox** as the encryption prov **Audit:** -``` bash +``` /bin/ps -ef | grep kube-apiserver | grep -v grep ``` @@ -1093,13 +1107,13 @@ on the master node and set the below parameter. **Audit:** -``` bash +``` /bin/ps -ef | grep kube-apiserver | grep -v grep ``` **Expected result**: -``` bash +``` 'TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256' has 'TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256' ``` @@ -1120,13 +1134,13 @@ for example: **Audit:** -``` bash +``` /bin/ps -ef | grep kube-controller-manager | grep -v grep ``` **Expected result**: -``` bash +``` '--terminated-pod-gc-threshold' is present ``` @@ -1144,13 +1158,13 @@ on the master node and set the below parameter. **Audit:** -``` bash +``` /bin/ps -ef | grep kube-controller-manager | grep -v grep ``` **Expected result**: -``` bash +``` 'false' is equal to 'false' ``` @@ -1168,13 +1182,13 @@ on the master node to set the below parameter. **Audit:** -``` bash +``` /bin/ps -ef | grep kube-controller-manager | grep -v grep ``` **Expected result**: -``` bash +``` 'true' is not equal to 'false' ``` @@ -1193,13 +1207,13 @@ to the private key file for service accounts. **Audit:** -``` bash +``` /bin/ps -ef | grep kube-controller-manager | grep -v grep ``` **Expected result**: -``` bash +``` '--service-account-private-key-file' is present ``` @@ -1217,13 +1231,13 @@ on the master node and set the `--root-ca-file` parameter to the certificate bun **Audit:** -``` bash +``` /bin/ps -ef | grep kube-controller-manager | grep -v grep ``` **Expected result**: -``` bash +``` '--root-ca-file' is present ``` @@ -1241,13 +1255,13 @@ on the master node and set the `--feature-gates` parameter to include `RotateKub **Audit:** -``` bash +``` /bin/ps -ef | grep kube-controller-manager | grep -v grep ``` **Expected result**: -``` bash +``` 'RotateKubeletServerCertificate=true' is equal to 'RotateKubeletServerCertificate=true' ``` @@ -1261,13 +1275,13 @@ on the master node and ensure the correct value for the `--bind-address` paramet **Audit:** -``` bash +``` /bin/ps -ef | grep kube-controller-manager | grep -v grep ``` **Expected result**: -``` bash +``` '--bind-address' is present OR '--bind-address' is not present ``` @@ -1287,13 +1301,13 @@ on the master node and set the below parameter. **Audit:** -``` bash +``` /bin/ps -ef | grep kube-scheduler | grep -v grep ``` **Expected result**: -``` bash +``` 'false' is equal to 'false' ``` @@ -1307,13 +1321,13 @@ on the master node and ensure the correct value for the `--bind-address` paramet **Audit:** -``` bash +``` /bin/ps -ef | grep kube-scheduler | grep -v grep ``` **Expected result**: -``` bash +``` '--bind-address' is present OR '--bind-address' is not present ``` @@ -1336,13 +1350,13 @@ on the master node and set the below parameters. **Audit:** -``` bash +``` /bin/ps -ef | /bin/grep etcd | /bin/grep -v grep ``` **Expected result**: -``` bash +``` '--cert-file' is present AND '--key-file' is present ``` @@ -1360,13 +1374,13 @@ node and set the below parameter. **Audit:** -``` bash +``` /bin/ps -ef | /bin/grep etcd | /bin/grep -v grep ``` **Expected result**: -``` bash +``` 'true' is equal to 'true' ``` @@ -1384,13 +1398,13 @@ node and either remove the `--auto-tls` parameter or set it to `false`. **Audit:** -``` bash +``` /bin/ps -ef | /bin/grep etcd | /bin/grep -v grep ``` **Expected result**: -``` bash +``` '--auto-tls' is not present OR '--auto-tls' is not present ``` @@ -1410,13 +1424,13 @@ master node and set the below parameters. **Audit:** -``` bash +``` /bin/ps -ef | /bin/grep etcd | /bin/grep -v grep ``` **Expected result**: -``` bash +``` '--peer-cert-file' is present AND '--peer-key-file' is present ``` @@ -1434,13 +1448,13 @@ node and set the below parameter. **Audit:** -``` bash +``` /bin/ps -ef | /bin/grep etcd | /bin/grep -v grep ``` **Expected result**: -``` bash +``` 'true' is equal to 'true' ``` @@ -1458,13 +1472,13 @@ node and either remove the `--peer-auto-tls` parameter or set it to `false`. **Audit:** -``` bash +``` /bin/ps -ef | /bin/grep etcd | /bin/grep -v grep ``` **Expected result**: -``` bash +``` '--peer-auto-tls' is not present OR '--peer-auto-tls' is present ``` @@ -1485,13 +1499,13 @@ master node and set the below parameter. **Audit:** -``` bash +``` /bin/ps -ef | /bin/grep etcd | /bin/grep -v grep ``` **Expected result**: -``` bash +``` '--trusted-ca-file' is present ``` @@ -1592,13 +1606,13 @@ chown root:root **Audit:** -``` bash +``` /bin/sh -c 'if test -e /etc/kubernetes/ssl/kube-ca.pem; then stat -c %U:%G /etc/kubernetes/ssl/kube-ca.pem; fi' ``` **Expected result**: -``` bash +``` 'root:root' is equal to 'root:root' ``` @@ -1642,19 +1656,19 @@ systemctl restart kubelet.service **Audit:** -``` bash +``` /bin/ps -fC kubelet ``` **Audit Config:** -``` bash +``` /bin/cat /var/lib/kubelet/config.yaml ``` **Expected result**: -``` bash +``` 'false' is equal to 'false' ``` @@ -1681,19 +1695,19 @@ systemctl restart kubelet.service **Audit:** -``` bash +``` /bin/ps -fC kubelet ``` **Audit Config:** -``` bash +``` /bin/cat /var/lib/kubelet/config.yaml ``` **Expected result**: -``` bash +``` 'Webhook' not have 'AlwaysAllow' ``` @@ -1721,19 +1735,19 @@ systemctl restart kubelet.service **Audit:** -``` bash +``` /bin/ps -fC kubelet ``` **Audit Config:** -``` bash +``` /bin/cat /var/lib/kubelet/config.yaml ``` **Expected result**: -``` bash +``` '--client-ca-file' is present ``` @@ -1760,19 +1774,19 @@ systemctl restart kubelet.service **Audit:** -``` bash +``` /bin/ps -fC kubelet ``` **Audit Config:** -``` bash +``` /bin/cat /var/lib/kubelet/config.yaml ``` **Expected result**: -``` bash +``` '0' is equal to '0' ``` @@ -1800,19 +1814,19 @@ systemctl restart kubelet.service **Audit:** -``` bash +``` /bin/ps -fC kubelet ``` **Audit Config:** -``` bash +``` /bin/cat /var/lib/kubelet/config.yaml ``` **Expected result**: -``` bash +``` '1800s' is not equal to '0' OR '--streaming-connection-idle-timeout' is not present ``` @@ -1839,19 +1853,19 @@ systemctl restart kubelet.service **Audit:** -``` bash +``` /bin/ps -fC kubelet ``` **Audit Config:** -``` bash +``` /bin/cat /var/lib/kubelet/config.yaml ``` **Expected result**: -``` bash +``` 'true' is equal to 'true' ``` @@ -1874,19 +1888,19 @@ systemctl restart kubelet.service **Audit:** -``` bash +``` /bin/ps -fC kubelet ``` **Audit Config:** -``` bash +``` /bin/cat /var/lib/kubelet/config.yaml ``` **Expected result**: -``` bash +``` 'true' is equal to 'true' OR '--make-iptables-util-chains' is not present ``` @@ -1907,7 +1921,7 @@ systemctl restart kubelet.service **Audit:** -``` bash +``` /bin/ps -fC kubelet ``` @@ -1929,19 +1943,19 @@ systemctl restart kubelet.service **Audit:** -``` bash +``` /bin/ps -fC kubelet ``` **Audit Config:** -``` bash +``` /bin/cat /var/lib/kubelet/config.yaml ``` **Expected result**: -``` bash +``` '0' is equal to '0' ``` @@ -1972,19 +1986,19 @@ systemctl restart kubelet.service **Audit:** -``` bash +``` /bin/ps -fC kubelet ``` **Audit Config:** -``` bash +``` /bin/cat /var/lib/kubelet/config.yaml ``` **Expected result**: -``` bash +``` '--rotate-certificates' is present OR '--rotate-certificates' is not present ``` @@ -2009,19 +2023,19 @@ systemctl restart kubelet.service **Audit:** -``` bash +``` /bin/ps -fC kubelet ``` **Audit Config:** -``` bash +``` /bin/cat /var/lib/kubelet/config.yaml ``` **Expected result**: -``` bash +``` 'true' is equal to 'true' ``` @@ -2054,19 +2068,19 @@ systemctl restart kubelet.service **Audit:** -``` bash +``` /bin/ps -fC kubelet ``` **Audit Config:** -``` bash +``` /bin/cat /var/lib/kubelet/config.yaml ``` **Expected result**: -``` bash +``` 'TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256' contains valid elements from 'TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256' ``` From f4b3b1a5a763dc996721bdc0715ac89dd981c1ea Mon Sep 17 00:00:00 2001 From: Nelson Roberts Date: Mon, 3 Feb 2020 16:12:55 -0700 Subject: [PATCH 04/14] EIO-247 update order for left nav --- content/rancher/v2.x/en/security/_index.md | 3 ++- content/rancher/v2.x/en/security/benchmark-2.1/_index.md | 2 +- content/rancher/v2.x/en/security/benchmark-2.2/_index.md | 2 +- content/rancher/v2.x/en/security/benchmark-2.3.3/_index.md | 2 +- content/rancher/v2.x/en/security/benchmark-2.3/_index.md | 2 +- content/rancher/v2.x/en/security/hardening-2.1/_index.md | 2 +- content/rancher/v2.x/en/security/hardening-2.2/_index.md | 2 +- content/rancher/v2.x/en/security/hardening-2.3.3/_index.md | 2 +- content/rancher/v2.x/en/security/hardening-2.3/_index.md | 2 +- 9 files changed, 10 insertions(+), 9 deletions(-) diff --git a/content/rancher/v2.x/en/security/_index.md b/content/rancher/v2.x/en/security/_index.md index 7f1c9ff3da6..ab97c117bbb 100644 --- a/content/rancher/v2.x/en/security/_index.md +++ b/content/rancher/v2.x/en/security/_index.md @@ -55,6 +55,7 @@ Each version of the hardening guide is intended to be used with specific version Hardening Guide Version | Rancher Version | CIS Benchmark Version | Kubernetes Version ------------------------|----------------|-----------------------|------------------ +[Hardening Guide v2.3.4]({{}}/rancher/v2.x/en/security/hardening-2.3.4/) | Rancher v2.3.4 | Benchmark v1.5 | Kubernetes v1.15 [Hardening Guide v2.3.3]({{}}/rancher/v2.x/en/security/hardening-2.3.3/) | Rancher v2.3.3 | Benchmark v1.4.1 | Kubernetes v1.14, v1.15, and v1.16 [Hardening Guide v2.3]({{}}/rancher/v2.x/en/security/hardening-2.3/) | Rancher v2.3.0-v2.3.2 | Benchmark v1.4.1 | Kubernetes v1.15 [Hardening Guide v2.2]({{}}/rancher/v2.x/en/security/hardening-2.2/) | Rancher v2.2.x | Benchmark v1.4.1 and 1.4.0 | Kubernetes v1.13 @@ -101,4 +102,4 @@ Rancher is committed to informing the community of security issues in our produc ### Security Tips and Best Practices -Our [best practices guide]({{}}/rancher/v2.x/en/best-practices/management/#tips-for-security) includes basic tips for increasing security in Rancher. \ No newline at end of file +Our [best practices guide]({{}}/rancher/v2.x/en/best-practices/management/#tips-for-security) includes basic tips for increasing security in Rancher. diff --git a/content/rancher/v2.x/en/security/benchmark-2.1/_index.md b/content/rancher/v2.x/en/security/benchmark-2.1/_index.md index faf84c76dfe..3808f116fd5 100644 --- a/content/rancher/v2.x/en/security/benchmark-2.1/_index.md +++ b/content/rancher/v2.x/en/security/benchmark-2.1/_index.md @@ -1,6 +1,6 @@ --- title: CIS Benchmark Rancher Self-Assessment Guide v2.1 -weight: 105 +weight: 109 --- This document is a companion to the Rancher v2.1 security hardening guide. The hardening guide provides prescriptive guidance for hardening a production installation of Rancher, and this benchmark guide is meant to help you evaluate the level of security of the hardened cluster against each control in the benchmark. diff --git a/content/rancher/v2.x/en/security/benchmark-2.2/_index.md b/content/rancher/v2.x/en/security/benchmark-2.2/_index.md index 0defa3142ef..8dee3cf5a95 100644 --- a/content/rancher/v2.x/en/security/benchmark-2.2/_index.md +++ b/content/rancher/v2.x/en/security/benchmark-2.2/_index.md @@ -1,6 +1,6 @@ --- title: CIS Benchmark Rancher Self-Assessment Guide v2.2 -weight: 104 +weight: 108 --- This document is a companion to the Rancher v2.2 security hardening guide. The hardening guide provides prescriptive guidance for hardening a production installation of Rancher, and this benchmark guide is meant to help you evaluate the level of security of the hardened cluster against each control in the benchmark. diff --git a/content/rancher/v2.x/en/security/benchmark-2.3.3/_index.md b/content/rancher/v2.x/en/security/benchmark-2.3.3/_index.md index 74d07855260..44086210d49 100644 --- a/content/rancher/v2.x/en/security/benchmark-2.3.3/_index.md +++ b/content/rancher/v2.x/en/security/benchmark-2.3.3/_index.md @@ -1,6 +1,6 @@ --- title: CIS Benchmark Rancher Self-Assessment Guide - Rancher v2.3.3 -weight: 103 +weight: 106 --- This document is a companion to the Rancher v2.3.3 security hardening guide. The hardening guide provides prescriptive guidance for hardening a production installation of Rancher, and this benchmark guide is meant to help you evaluate the level of security of the hardened cluster against each control in the benchmark. diff --git a/content/rancher/v2.x/en/security/benchmark-2.3/_index.md b/content/rancher/v2.x/en/security/benchmark-2.3/_index.md index 7a0d5193ffb..74ff4c693a1 100644 --- a/content/rancher/v2.x/en/security/benchmark-2.3/_index.md +++ b/content/rancher/v2.x/en/security/benchmark-2.3/_index.md @@ -1,6 +1,6 @@ --- title: CIS Benchmark Rancher Self-Assessment Guide v2.3 -weight: 103 +weight: 107 --- This document is a companion to the Rancher v2.3 security hardening guide. The hardening guide provides prescriptive guidance for hardening a production installation of Rancher, and this benchmark guide is meant to help you evaluate the level of security of the hardened cluster against each control in the benchmark. diff --git a/content/rancher/v2.x/en/security/hardening-2.1/_index.md b/content/rancher/v2.x/en/security/hardening-2.1/_index.md index 5e027ab8c62..f7f1fcbf0e9 100644 --- a/content/rancher/v2.x/en/security/hardening-2.1/_index.md +++ b/content/rancher/v2.x/en/security/hardening-2.1/_index.md @@ -1,6 +1,6 @@ --- title: Hardening Guide v2.1 -weight: 102 +weight: 104 --- This document provides prescriptive guidance for hardening a production installation of Rancher v2.1.x. It outlines the configurations and controls required to address Kubernetes benchmark controls from the Center for Information Security (CIS). diff --git a/content/rancher/v2.x/en/security/hardening-2.2/_index.md b/content/rancher/v2.x/en/security/hardening-2.2/_index.md index 81104b9117f..201c778fff6 100644 --- a/content/rancher/v2.x/en/security/hardening-2.2/_index.md +++ b/content/rancher/v2.x/en/security/hardening-2.2/_index.md @@ -1,6 +1,6 @@ --- title: Hardening Guide v2.2 -weight: 101 +weight: 103 --- This document provides prescriptive guidance for hardening a production installation of Rancher v2.2.x. It outlines the configurations and controls required to address Kubernetes benchmark controls from the Center for Information Security (CIS). diff --git a/content/rancher/v2.x/en/security/hardening-2.3.3/_index.md b/content/rancher/v2.x/en/security/hardening-2.3.3/_index.md index ebc2a6ae8e9..bf6a72b0009 100644 --- a/content/rancher/v2.x/en/security/hardening-2.3.3/_index.md +++ b/content/rancher/v2.x/en/security/hardening-2.3.3/_index.md @@ -1,6 +1,6 @@ --- title: Hardening Guide v2.3.3 -weight: 100 +weight: 101 --- This document provides prescriptive guidance for hardening a production installation of Rancher v2.3.3. It outlines the configurations and controls required to address Kubernetes benchmark controls from the Center for Information Security (CIS). diff --git a/content/rancher/v2.x/en/security/hardening-2.3/_index.md b/content/rancher/v2.x/en/security/hardening-2.3/_index.md index 163c948a123..04336c2acaa 100644 --- a/content/rancher/v2.x/en/security/hardening-2.3/_index.md +++ b/content/rancher/v2.x/en/security/hardening-2.3/_index.md @@ -1,6 +1,6 @@ --- title: Hardening Guide v2.3 -weight: 100 +weight: 102 --- This document provides prescriptive guidance for hardening a production installation of Rancher v2.3.0-v2.3.2. It outlines the configurations and controls required to address Kubernetes benchmark controls from the Center for Information Security (CIS). From 8bba4e443fb66cb128463d523b8cd3655bb93209 Mon Sep 17 00:00:00 2001 From: Nelson Roberts Date: Wed, 12 Feb 2020 16:00:58 -0700 Subject: [PATCH 05/14] EIO-246: add 2.3.5, update hardening guide and self-assessment --- content/rancher/v2.x/en/security/_index.md | 2 +- .../_index.md | 730 +++++------------- .../en/security/hardening-2.3.4/_index.md | 491 ------------ .../en/security/hardening-2.3.5/_index.md | 424 ++++++++++ 4 files changed, 622 insertions(+), 1025 deletions(-) rename content/rancher/v2.x/en/security/{benchmark-2.3.4 => benchmark-2.3.5}/_index.md (73%) delete mode 100644 content/rancher/v2.x/en/security/hardening-2.3.4/_index.md create mode 100644 content/rancher/v2.x/en/security/hardening-2.3.5/_index.md diff --git a/content/rancher/v2.x/en/security/_index.md b/content/rancher/v2.x/en/security/_index.md index ab97c117bbb..4789b369162 100644 --- a/content/rancher/v2.x/en/security/_index.md +++ b/content/rancher/v2.x/en/security/_index.md @@ -55,7 +55,7 @@ Each version of the hardening guide is intended to be used with specific version Hardening Guide Version | Rancher Version | CIS Benchmark Version | Kubernetes Version ------------------------|----------------|-----------------------|------------------ -[Hardening Guide v2.3.4]({{}}/rancher/v2.x/en/security/hardening-2.3.4/) | Rancher v2.3.4 | Benchmark v1.5 | Kubernetes v1.15 +[Hardening Guide v2.3.5]({{}}/rancher/v2.x/en/security/hardening-2.3.5/) | Rancher v2.3.5 | Benchmark v1.5 | Kubernetes v1.15 [Hardening Guide v2.3.3]({{}}/rancher/v2.x/en/security/hardening-2.3.3/) | Rancher v2.3.3 | Benchmark v1.4.1 | Kubernetes v1.14, v1.15, and v1.16 [Hardening Guide v2.3]({{}}/rancher/v2.x/en/security/hardening-2.3/) | Rancher v2.3.0-v2.3.2 | Benchmark v1.4.1 | Kubernetes v1.15 [Hardening Guide v2.2]({{}}/rancher/v2.x/en/security/hardening-2.2/) | Rancher v2.2.x | Benchmark v1.4.1 and 1.4.0 | Kubernetes v1.13 diff --git a/content/rancher/v2.x/en/security/benchmark-2.3.4/_index.md b/content/rancher/v2.x/en/security/benchmark-2.3.5/_index.md similarity index 73% rename from content/rancher/v2.x/en/security/benchmark-2.3.4/_index.md rename to content/rancher/v2.x/en/security/benchmark-2.3.5/_index.md index 57e25c15e7c..d06ce9b3af0 100644 --- a/content/rancher/v2.x/en/security/benchmark-2.3.4/_index.md +++ b/content/rancher/v2.x/en/security/benchmark-2.3.5/_index.md @@ -1,23 +1,23 @@ --- -title: CIS Benchmark Rancher Self-Assessment Guide - Rancher v2.3.4 +title: CIS Benchmark Rancher Self-Assessment Guide - v2.3.5 weight: 105 --- -### CIS Kubernetes Benchmark 1.5 - Rancher 2.3.4 with Kubernetes 1.15 +### CIS Kubernetes Benchmark 1.5 - Rancher 2.3.5 with Kubernetes 1.15 -[Click here to download a PDF version of this document](https://releases.rancher.com/documents/security/2.3.4/Rancher_Benchmark_Assessment.pdf) +[Click here to download a PDF version of this document](https://releases.rancher.com/documents/security/2.3.5/Rancher_Benchmark_Assessment.pdf) #### Overview -This document is a companion to the Rancher v2.3.4 security hardening guide. The hardening guide provides prescriptive guidance for hardening a production installation of Rancher, and this benchmark guide is meant to help you evaluate the level of security of the hardened cluster against each control in the benchmark. +This document is a companion to the Rancher v2.3.5 security hardening guide. The hardening guide provides prescriptive guidance for hardening a production installation of Rancher, and this benchmark guide is meant to help you evaluate the level of security of the hardened cluster against each control in the benchmark. This guide corresponds to specific versions of the hardening guide, Rancher, Kubernetes, and the CIS Benchmark: Self Assessment Guide Version | Rancher Version | Hardening Guide Version | Kubernetes Version | CIS Benchmark Version ---------------------------|----------|---------|-------|----- -Self Assessment Guide v2.3.4 | Rancher v2.3.4 | Hardening Guide v2.3.4 | Kubernetes v1.15 | Benchmark v1.5 +Self Assessment Guide v2.3.5 | Rancher v2.3.5 | Hardening Guide v2.3.5 | Kubernetes v1.15 | Benchmark v1.5 -Because Rancher and RKE install Kubernetes services as Docker containers, many of the control verification checks in the CIS Kubernetes Benchmark don't apply. This guide will walk through the various controls and provide updated example commands to audit compliance in Rancher-created clusters. +Because Rancher and RKE install Kubernetes services as Docker containers, many of the control verification checks in the CIS Kubernetes Benchmark don't apply and will have a result of `Not Applicable`. This guide will walk through the various controls and provide updated example commands to audit compliance in Rancher-created clusters. This document is to be used by Rancher operators, security teams, auditors and decision makers. @@ -27,9 +27,9 @@ For more detail about each audit, including rationales and remediations for fail Rancher and RKE install Kubernetes services via Docker containers. Configuration is defined by arguments passed to the container at the time of initialization, not via configuration files. -Scoring the commands is different in Rancher Labs than in the CIS Benchmark. Where the commands differ from the original CIS benchmark, the commands specific to Rancher Labs are provided for testing. +Scoring the commands is different in Rancher Labs than in the CIS Benchmark. Where the commands differ from the original CIS benchmark, the commands specific to Rancher Labs are provided for testing. Only **scored** tests will be covered in this guide. -When performing the tests, you will need access to the Docker command line on the hosts of all three RKE roles. The commands also make use of the the `jq` command to provide human-readable formatting. +When performing the tests, you will need access to the Docker command line on the hosts of all three RKE roles. The commands also make use of the the `jq` and `kubectl` (with valid config) commands to provide human-readable formatting. ### Controls @@ -39,96 +39,60 @@ When performing the tests, you will need access to the Docker command line on th #### 1.1.1 Ensure that the API server pod specification file permissions are set to `644` or more restrictive (Scored) -**Result:** INFO +**Result:** Not Applicable **Remediation:** RKE doesn’t require or maintain a configuration file for the API server. All configuration is passed in as arguments at container run time. #### 1.1.2 Ensure that the API server pod specification file ownership is set to `root:root` (Scored) -**Result:** INFO +**Result:** Not Applicable **Remediation:** RKE doesn’t require or maintain a configuration file for the API server. All configuration is passed in as arguments at container run time. #### 1.1.3 Ensure that the controller manager pod specification file permissions are set to `644` or more restrictive (Scored) -**Result:** INFO +**Result:** Not Applicable **Remediation:** RKE doesn’t require or maintain a configuration file for the controller manager. All configuration is passed in as arguments at container run time. #### 1.1.4 Ensure that the controller manager pod specification file ownership is set to `root:root` (Scored) -**Result:** INFO +**Result:** Not Applicable **Remediation:** RKE doesn’t require or maintain a configuration file for the controller manager. All configuration is passed in as arguments at container run time. #### 1.1.5 Ensure that the scheduler pod specification file permissions are set to `644` or more restrictive (Scored) -**Result:** INFO +**Result:** Not Applicable **Remediation:** RKE doesn’t require or maintain a configuration file for the scheduler. All configuration is passed in as arguments at container run time. #### 1.1.6 Ensure that the scheduler pod specification file ownership is set to `root:root` (Scored) -**Result:** INFO +**Result:** Not Applicable **Remediation:** RKE doesn’t require or maintain a configuration file for the scheduler. All configuration is passed in as arguments at container run time. #### 1.1.7 Ensure that the etcd pod specification file permissions are set to `644` or more restrictive (Scored) -**Result:** INFO +**Result:** Not Applicable **Remediation:** RKE doesn’t require or maintain a configuration file for etcd. All configuration is passed in as arguments at container run time. #### 1.1.8 Ensure that the etcd pod specification file ownership is set to `root:root` (Scored) -**Result:** INFO +**Result:** Not Applicable **Remediation:** RKE doesn’t require or maintain a configuration file for etcd. All configuration is passed in as arguments at container run time. -#### 1.1.9 Ensure that the Container Network Interface file permissions are set to `644` or more restrictive (Not Scored) - -**Result:** WARN - -**Remediation:** -Run the below command (based on the file location on your system) on the master node. -For example, - -``` bash -chmod 644 -``` - -**Audit:** - -``` -stat -c %a -``` - -#### 1.1.10 Ensure that the Container Network Interface file ownership is set to `root:root` (Not Scored) - -**Result:** WARN - -**Remediation:** -Run the below command (based on the file location on your system) on the master node. -For example, - -``` bash -chown root:root -``` - -**Audit:** - -``` -stat -c %U:%G -``` - #### 1.1.11 Ensure that the etcd data directory permissions are set to `700` or more restrictive (Scored) **Result:** PASS @@ -150,7 +114,7 @@ chmod 700 /var/lib/etcd **Audit:** ``` -ps -ef | grep etcd | grep -- --data-dir | sed 's%.*data-dir[= ]\([^ ]*\).*%\1%' | xargs stat -c %a +/mnt/kube-bench/test_helpers/1.1.11.sh etcd ``` **Expected result**: @@ -173,14 +137,14 @@ ps -ef | grep etcd Run the below command (based on the etcd data directory found above). For example, -``` bash +``` bash chown etcd:etcd /var/lib/etcd ``` **Audit:** ``` -ps -ef | grep etcd | grep -- --data-dir | sed 's%.*data-dir[= ]\([^ ]*\).*%\1%' | xargs stat -c %U:%G +/mnt/kube-bench/test_helpers/1.1.12.sh etcd ``` **Expected result**: @@ -191,7 +155,7 @@ ps -ef | grep etcd | grep -- --data-dir | sed 's%.*data-dir[= ]\([^ ]*\).*%\1%' #### 1.1.13 Ensure that the `admin.conf` file permissions are set to `644` or more restrictive (Scored) -**Result:** INFO +**Result:** Not Applicable **Remediation:** RKE does not store the kubernetes default kubeconfig credentials file on the nodes. It’s presented to user where RKE is run. @@ -199,7 +163,7 @@ We recommend that this `kube_config_cluster.yml` file be kept in secure store. #### 1.1.14 Ensure that the admin.conf file ownership is set to `root:root` (Scored) -**Result:** INFO +**Result:** Not Applicable **Remediation:** RKE does not store the kubernetes default kubeconfig credentials file on the nodes. It’s presented to user where RKE is run. @@ -207,112 +171,106 @@ We recommend that this `kube_config_cluster.yml` file be kept in secure store. #### 1.1.15 Ensure that the `scheduler.conf` file permissions are set to `644` or more restrictive (Scored) -**Result:** INFO +**Result:** Not Applicable **Remediation:** RKE doesn’t require or maintain a configuration file for the scheduler. All configuration is passed in as arguments at container run time. #### 1.1.16 Ensure that the `scheduler.conf` file ownership is set to `root:root` (Scored) -**Result:** INFO +**Result:** Not Applicable **Remediation:** RKE doesn’t require or maintain a configuration file for the scheduler. All configuration is passed in as arguments at container run time. #### 1.1.17 Ensure that the `controller-manager.conf` file permissions are set to `644` or more restrictive (Scored) -**Result:** INFO +**Result:** Not Applicable **Remediation:** RKE doesn’t require or maintain a configuration file for the controller manager. All configuration is passed in as arguments at container run time. #### 1.1.18 Ensure that the `controller-manager.conf` file ownership is set to `root:root` (Scored) -**Result:** INFO +**Result:** Not Applicable **Remediation:** RKE doesn’t require or maintain a configuration file for the controller manager. All configuration is passed in as arguments at container run time. #### 1.1.19 Ensure that the Kubernetes PKI directory and file ownership is set to `root:root` (Scored) -**Result:** WARN - -**Remediation:** -Run the below command (based on the file location on your system) on the master node. -For example, - -``` bash -chown -R root:root /etc/kubernetes/pki/ -``` - -**Audit:** - -``` -ls -laR /etc/kubernetes/pki/ -``` - -#### 1.1.20 Ensure that the Kubernetes PKI certificate file permissions are set to `644` or more restrictive (Scored) - -**Result:** WARN - -**Remediation:** -Run the below command (based on the file location on your system) on the master node. -For example, - -``` bash -chmod -R 644 /etc/kubernetes/pki/*.crt -``` - -**Audit:** - -``` -stat -c %n %a /etc/kubernetes/pki/*.crt -``` - -#### 1.1.21 Ensure that the Kubernetes PKI key file permissions are set to `600` (Scored) - -**Result:** WARN - -**Remediation:** -Run the below command (based on the file location on your system) on the master node. -For example, - -``` bash -chmod -R 600 /etc/kubernetes/pki/*.key -``` - -**Audit:** - -``` -stat -c %n %a /etc/kubernetes/pki/*.key -``` - -### 1.2 API Server - -#### 1.2.1 Ensure that the `--anonymous-auth` argument is set to `false` (Not Scored) - **Result:** PASS **Remediation:** -Edit the API server pod specification file `/etc/kubernetes/manifests/kube-apiserver.yaml` -on the master node and set the below parameter. +Run the below command (based on the file location on your system) on the master node. +For example, ``` bash ---anonymous-auth=false +chown -R root:root /etc/kubernetes/ssl ``` **Audit:** ``` -/bin/ps -ef | grep kube-apiserver | grep -v grep +stat -c %U:%G /etc/kubernetes/ssl ``` **Expected result**: ``` -'false' is equal to 'false' +'root:root' is present ``` +#### 1.1.20 Ensure that the Kubernetes PKI certificate file permissions are set to `644` or more restrictive (Scored) + +**Result:** PASS + +**Remediation:** +Run the below command (based on the file location on your system) on the master node. +For example, + +``` bash +chmod -R 644 /etc/kubernetes/ssl" +``` + +**Audit:** + +``` +/mnt/kube-bench/test_helpers/check_files_permissions.sh '/etc/kubernetes/ssl/*.pem' +``` + +**Expected result**: + +``` +'true' is present +``` + +#### 1.1.21 Ensure that the Kubernetes PKI key file permissions are set to `600` (Scored) + +**Result:** PASS + +**Remediation:** +Run the below command (based on the file location on your system) on the master node. +For example, + +``` bash +chmod -R 600 /etc/kubernetes/ssl/certs/serverca +``` + +**Audit:** + +``` +/mnt/kube-bench/test_helpers/1.1.21.sh /etc/kubernetes/ssl +``` + +**Expected result**: + +``` +'pass' is present +``` + +### 1.2 API Server + #### 1.2.2 Ensure that the `--basic-auth-file` argument is not set (Scored) **Result:** PASS @@ -499,32 +457,6 @@ for example: 'Node,RBAC' has 'RBAC' ``` -#### 1.2.10 Ensure that the admission control plugin `EventRateLimit` is set (Not Scored) - -**Result:** PASS - -**Remediation:** -Follow the Kubernetes documentation and set the desired limits in a configuration file. -Then, edit the API server pod specification file `/etc/kubernetes/manifests/kube-apiserver.yaml` -and set the below parameters. - -``` bash ---enable-admission-plugins=...,EventRateLimit,... ---admission-control-config-file= -``` - -**Audit:** - -``` -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected result**: - -``` -'ServiceAccount,NamespaceLifecycle,LimitRanger,PersistentVolumeLabel,DefaultStorageClass,ResourceQuota,DefaultTolerationSeconds,AlwaysPullImages,DenyEscalatingExec,NodeRestriction,EventRateLimit,PodSecurityPolicy' has 'EventRateLimit' -``` - #### 1.2.11 Ensure that the admission control plugin `AlwaysAdmit` is not set (Scored) **Result:** PASS @@ -543,51 +475,7 @@ value that does not include `AlwaysAdmit`. **Expected result**: ``` -'ServiceAccount,NamespaceLifecycle,LimitRanger,PersistentVolumeLabel,DefaultStorageClass,ResourceQuota,DefaultTolerationSeconds,AlwaysPullImages,DenyEscalatingExec,NodeRestriction,EventRateLimit,PodSecurityPolicy' not have 'AlwaysAdmit' OR '--enable-admission-plugins' is not present -``` - -#### 1.2.12 Ensure that the admission control plugin `AlwaysPullImages` is set (Not Scored) - -**Result:** PASS - -**Remediation:** -Edit the API server pod specification file `/etc/kubernetes/manifests/kube-apiserver.yaml` -on the master node and set the `--enable-admission-plugins` parameter to include -`AlwaysPullImages`. - -``` bash ---enable-admission-plugins=...,AlwaysPullImages,... -``` - -**Audit:** - -``` -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected result**: - -``` -'ServiceAccount,NamespaceLifecycle,LimitRanger,PersistentVolumeLabel,DefaultStorageClass,ResourceQuota,DefaultTolerationSeconds,AlwaysPullImages,DenyEscalatingExec,NodeRestriction,EventRateLimit,PodSecurityPolicy' has 'AlwaysPullImages' -``` - -#### 1.2.13 Ensure that the admission control plugin `SecurityContextDeny` is set if `PodSecurityPolicy` is not used (Not Scored) - -**Result:** WARN - -**Remediation:** -Edit the API server pod specification file `/etc/kubernetes/manifests/kube-apiserver.yaml` -on the master node and set the `--enable-admission-plugins` parameter to include -`SecurityContextDeny`, unless `PodSecurityPolicy` is already in place. - -``` bash ---enable-admission-plugins=...,SecurityContextDeny,... -``` - -**Audit:** - -``` -/bin/ps -ef | grep kube-apiserver | grep -v grep +'NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit' not have 'AlwaysAdmit' OR '--enable-admission-plugins' is not present ``` #### 1.2.14 Ensure that the admission control plugin `ServiceAccount` is set (Scored) @@ -609,7 +497,7 @@ value that does not include `ServiceAccount`. **Expected result**: ``` -'ServiceAccount,NamespaceLifecycle,LimitRanger,PersistentVolumeLabel,DefaultStorageClass,ResourceQuota,DefaultTolerationSeconds,AlwaysPullImages,DenyEscalatingExec,NodeRestriction,EventRateLimit,PodSecurityPolicy' has 'ServiceAccount' OR '--enable-admission-plugins' is not present +'NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit' has 'ServiceAccount' OR '--enable-admission-plugins' is not present ``` #### 1.2.15 Ensure that the admission control plugin `NamespaceLifecycle` is set (Scored) @@ -658,7 +546,7 @@ Then restart the API Server. **Expected result**: ``` -'ServiceAccount,NamespaceLifecycle,LimitRanger,PersistentVolumeLabel,DefaultStorageClass,ResourceQuota,DefaultTolerationSeconds,AlwaysPullImages,DenyEscalatingExec,NodeRestriction,EventRateLimit,PodSecurityPolicy' has 'PodSecurityPolicy' +'NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit' has 'PodSecurityPolicy' ``` #### 1.2.17 Ensure that the admission control plugin `NodeRestriction` is set (Scored) @@ -684,7 +572,7 @@ value that includes `NodeRestriction`. **Expected result**: ``` -'ServiceAccount,NamespaceLifecycle,LimitRanger,PersistentVolumeLabel,DefaultStorageClass,ResourceQuota,DefaultTolerationSeconds,AlwaysPullImages,DenyEscalatingExec,NodeRestriction,EventRateLimit,PodSecurityPolicy' has 'NodeRestriction' +'NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit' has 'NodeRestriction' ``` #### 1.2.18 Ensure that the `--insecure-bind-address` argument is not set (Scored) @@ -1081,7 +969,7 @@ on the master node and set the `--encryption-provider-config` parameter to the p #### 1.2.34 Ensure that encryption providers are appropriately configured (Scored) -**Result:** WARN +**Result:** PASS **Remediation:** Follow the Kubernetes documentation and configure a `EncryptionConfig` file. @@ -1090,31 +978,13 @@ In this file, choose **aescbc**, **kms** or **secretbox** as the encryption prov **Audit:** ``` -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -#### 1.2.35 Ensure that the API Server only makes use of Strong Cryptographic Ciphers (Not Scored) - -**Result:** PASS - -**Remediation:** -Edit the API server pod specification file `/etc/kubernetes/manifests/kube-apiserver.yaml` -on the master node and set the below parameter. - -``` bash ---tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 -``` - -**Audit:** - -``` -/bin/ps -ef | grep kube-apiserver | grep -v grep +/mnt/kube-bench/test_helpers/1.2.34.sh /etc/kubernetes/ssl/encryption.yaml ``` **Expected result**: ``` -'TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256' has 'TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256' +'--pass' is present ``` ### 1.3 Controller Manager @@ -1482,109 +1352,76 @@ node and either remove the `--peer-auto-tls` parameter or set it to `false`. '--peer-auto-tls' is not present OR '--peer-auto-tls' is present ``` -#### 2.7 Ensure that a unique Certificate Authority is used for etcd (Not Scored) +## 3 Control Plane Configuration +### 3.2 Logging + +#### 3.2.1 Ensure that a minimal audit policy is created (Scored) **Result:** PASS **Remediation:** -[Manual test] -Follow the etcd documentation and create a dedicated certificate authority setup for the -etcd service. -Then, edit the etcd pod specification file `/etc/kubernetes/manifests/etcd.yaml` on the -master node and set the below parameter. - -``` bash ---trusted-ca-file= -``` +Create an audit policy file for your cluster. **Audit:** ``` -/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep +/mnt/kube-bench/test_helpers/3.2.1.sh kube-apiserver ``` **Expected result**: ``` -'--trusted-ca-file' is present +'--audit-policy-file' is present ``` -## 3 Control Plane Configuration -### 3.1 Authentication and Authorization - -#### 3.1.1 Client certificate authentication should not be used for users (Not Scored) - -**Result:** WARN - -**Remediation:** -Alternative mechanisms provided by Kubernetes such as the use of OIDC should be -implemented in place of client certificates. - -### 3.2 Logging - -#### 3.2.1 Ensure that a minimal audit policy is created (Scored) - -**Result:** WARN - -**Remediation:** -Create an audit policy file for your cluster. - -#### 3.2.2 Ensure that the audit policy covers key security concerns (Not Scored) - -**Result:** WARN - -**Remediation:** -Consider modification of the audit policy in use on the cluster to include these items, at a -minimum. - ## 4 Worker Node Security Configuration ### 4.1 Worker Node Configuration Files #### 4.1.1 Ensure that the kubelet service file permissions are set to `644` or more restrictive (Scored) -**Result:** INFO +**Result:** Not Applicable **Remediation:** RKE doesn’t require or maintain a configuration file for the kubelet service. All configuration is passed in as arguments at container run time. #### 4.1.2 Ensure that the kubelet service file ownership is set to `root:root` (Scored) -**Result:** INFO +**Result:** Not Applicable **Remediation:** RKE doesn’t require or maintain a configuration file for the kubelet service. All configuration is passed in as arguments at container run time. #### 4.1.3 Ensure that the proxy kubeconfig file permissions are set to `644` or more restrictive (Scored) -**Result:** INFO +**Result:** Not Applicable **Remediation:** RKE doesn’t require or maintain a configuration file for the proxy service. All configuration is passed in as arguments at container run time. #### 4.1.4 Ensure that the proxy kubeconfig file ownership is set to `root:root` (Scored) -**Result:** INFO +**Result:** Not Applicable **Remediation:** RKE doesn’t require or maintain a configuration file for the proxy service. All configuration is passed in as arguments at container run time. #### 4.1.5 Ensure that the kubelet.conf file permissions are set to `644` or more restrictive (Scored) -**Result:** INFO +**Result:** Not Applicable **Remediation:** RKE doesn’t require or maintain a configuration file for the kubelet service. All configuration is passed in as arguments at container run time. #### 4.1.6 Ensure that the kubelet.conf file ownership is set to `root:root` (Scored) -**Result:** INFO +**Result:** Not Applicable **Remediation:** RKE doesn’t require or maintain a configuration file for the kubelet service. All configuration is passed in as arguments at container run time. #### 4.1.7 Ensure that the certificate authorities file permissions are set to `644` or more restrictive (Scored) -**Result:** WARN +**Result:** PASS **Remediation:** Run the following command to modify the file permissions of the @@ -1593,6 +1430,18 @@ Run the following command to modify the file permissions of the --client-ca-file chmod 644 ``` +**Audit:** + +``` +stat -c %a /etc/kubernetes/ssl/kube-ca.pem +``` + +**Expected result**: + +``` +'644' is equal to '644' OR '640' is present OR '600' is present +``` + #### 4.1.8 Ensure that the client certificate authorities file ownership is set to `root:root` (Scored) **Result:** PASS @@ -1618,14 +1467,14 @@ chown root:root #### 4.1.9 Ensure that the kubelet configuration file has permissions set to `644` or more restrictive (Scored) -**Result:** INFO +**Result:** Not Applicable **Remediation:** RKE doesn’t require or maintain a configuration file for the kubelet service. All configuration is passed in as arguments at container run time. #### 4.1.10 Ensure that the kubelet configuration file ownership is set to `root:root` (Scored) -**Result:** INFO +**Result:** Not Applicable **Remediation:** RKE doesn’t require or maintain a configuration file for the kubelet service. All configuration is passed in as arguments at container run time. @@ -1827,7 +1676,7 @@ systemctl restart kubelet.service **Expected result**: ``` -'1800s' is not equal to '0' OR '--streaming-connection-idle-timeout' is not present +'30m' is not equal to '0' OR '--streaming-connection-idle-timeout' is not present ``` #### 4.2.6 Ensure that the ```--protect-kernel-defaults``` argument is set to `true` (Scored) @@ -1904,64 +1753,9 @@ systemctl restart kubelet.service 'true' is equal to 'true' OR '--make-iptables-util-chains' is not present ``` -#### 4.2.8 Ensure that the `--hostname-override` argument is not set (Not Scored) - -**Result:** WARN - -**Remediation:** -Edit the kubelet service file `/etc/systemd/system/kubelet.service.d/10-kubeadm.conf` -on each worker node and remove the `--hostname-override` argument from the -`KUBELET_SYSTEM_PODS_ARGS` variable. -Based on your system, restart the kubelet service. For example: - -``` bash -systemctl daemon-reload -systemctl restart kubelet.service -``` - -**Audit:** - -``` -/bin/ps -fC kubelet -``` - -#### 4.2.9 Ensure that the `--event-qps` argument is set to `0` or a level which ensures appropriate event capture (Not Scored) - -**Result:** PASS - -**Remediation:** -If using a Kubelet config file, edit the file to set `eventRecordQPS`: to an appropriate level. -If using command line arguments, edit the kubelet service file -`/etc/systemd/system/kubelet.service.d/10-kubeadm.conf` on each worker node and -set the below parameter in `KUBELET_SYSTEM_PODS_ARGS` variable. -Based on your system, restart the kubelet service. For example: - -``` bash -systemctl daemon-reload -systemctl restart kubelet.service -``` - -**Audit:** - -``` -/bin/ps -fC kubelet -``` - -**Audit Config:** - -``` -/bin/cat /var/lib/kubelet/config.yaml -``` - -**Expected result**: - -``` -'0' is equal to '0' -``` - #### 4.2.10 Ensure that the `--tls-cert-file` and `--tls-private-key-file` arguments are set as appropriate (Scored) -**Result:** INFO +**Result:** Not Applicable **Remediation:** RKE doesn’t require or maintain a configuration file for the kubelet service. All configuration is passed in as arguments at container run time. @@ -2039,90 +1833,12 @@ systemctl restart kubelet.service 'true' is equal to 'true' ``` -#### 4.2.13 Ensure that the Kubelet only makes use of Strong Cryptographic Ciphers (Not Scored) - -**Result:** PASS - -**Remediation:** -If using a Kubelet config file, edit the file to set `TLSCipherSuites`: to - -``` bash -TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 -``` - -or to a subset of these values. -If using executable arguments, edit the kubelet service file -`/etc/systemd/system/kubelet.service.d/10-kubeadm.conf` on each worker node and -set the `--tls-cipher-suites` parameter as follows, or to a subset of these values. - -``` bash ---tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 -``` - -Based on your system, restart the kubelet service. For example: - -``` bash -systemctl daemon-reload -systemctl restart kubelet.service -``` - -**Audit:** - -``` -/bin/ps -fC kubelet -``` - -**Audit Config:** - -``` -/bin/cat /var/lib/kubelet/config.yaml -``` - -**Expected result**: - -``` -'TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256' contains valid elements from 'TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256' -``` - ## 5 Kubernetes Policies ### 5.1 RBAC and Service Accounts -#### 5.1.1 Ensure that the cluster-admin role is only used where required (Not Scored) - -**Result:** WARN - -**Remediation:** -Identify all `clusterrolebindings` to the `cluster-admin` role. Check if they are used and -if they need this role or if they could use a role with fewer privileges. -Where possible, first bind users to a lower privileged role and then remove the -`clusterrolebinding` to the `cluster-admin` role : - -``` bash -kubectl delete clusterrolebinding [name] -``` - -#### 5.1.2 Minimize access to secrets (Not Scored) - -**Result:** WARN - -**Remediation:** -Where possible, remove `get`, `list` and `watch` access to secret objects in the cluster. - -#### 5.1.3 Minimize wildcard use in Roles and ClusterRoles (Not Scored) - -**Result:** WARN - -**Remediation:** -Where possible replace any use of wildcards in `clusterroles` and roles with specific -objects or actions. - -#### 5.1.4 Minimize access to create pods (Not Scored) - -**Result:** WARN - #### 5.1.5 Ensure that default service accounts are not actively used. (Scored) -**Result:** WARN +**Result:** PASS **Remediation:** Create explicit service accounts wherever a Kubernetes workload requires specific access @@ -2133,192 +1849,140 @@ Modify the configuration of each default service account to include this value automountServiceAccountToken: false ``` -#### 5.1.6 Ensure that Service Account Tokens are only mounted where necessary (Not Scored) +**Audit:** -**Result:** WARN +``` +/mnt/kube-bench/test_helpers/5.1.5.sh +``` -**Remediation:** -Modify the definition of pods and service accounts which do not need to mount service -account tokens to disable it. +**Expected result**: + +``` +'--pass' is present +``` ### 5.2 Pod Security Policies -#### 5.2.1 Minimize the admission of privileged containers (Not Scored) - -**Result:** WARN - -**Remediation:** -Create a PSP as described in the Kubernetes documentation, ensuring that -the `.spec.privileged` field is omitted or set to `false`. - #### 5.2.2 Minimize the admission of containers wishing to share the host process ID namespace (Scored) -**Result:** WARN +**Result:** PASS **Remediation:** Create a PSP as described in the Kubernetes documentation, ensuring that the `.spec.hostPID` field is omitted or set to `false`. +**Audit:** + +``` +kubectl --kubeconfig=/root/.kube/config get psp -o json | jq .items[] | jq -r 'select((.spec.hostPID == null) or (.spec.hostPID == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' +``` + +**Expected result**: + +``` +1 is greater than 0 +``` + #### 5.2.3 Minimize the admission of containers wishing to share the host IPC namespace (Scored) -**Result:** WARN +**Result:** PASS **Remediation:** Create a PSP as described in the Kubernetes documentation, ensuring that the `.spec.hostIPC` field is omitted or set to `false`. +**Audit:** + +``` +kubectl --kubeconfig=/root/.kube/config get psp -o json | jq .items[] | jq -r 'select((.spec.hostIPC == null) or (.spec.hostIPC == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' +``` + +**Expected result**: + +``` +1 is greater than 0 +``` + #### 5.2.4 Minimize the admission of containers wishing to share the host network namespace (Scored) -**Result:** WARN +**Result:** PASS **Remediation:** Create a PSP as described in the Kubernetes documentation, ensuring that the `.spec.hostNetwork` field is omitted or set to `false`. +**Audit:** + +``` +kubectl --kubeconfig=/root/.kube/config get psp -o json | jq .items[] | jq -r 'select((.spec.hostNetwork == null) or (.spec.hostNetwork == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' +``` + +**Expected result**: + +``` +1 is greater than 0 +``` + #### 5.2.5 Minimize the admission of containers with `allowPrivilegeEscalation` (Scored) -**Result:** WARN +**Result:** PASS **Remediation:** Create a PSP as described in the Kubernetes documentation, ensuring that the `.spec.allowPrivilegeEscalation` field is omitted or set to `false`. -#### 5.2.6 Minimize the admission of root containers (Not Scored) +**Audit:** -**Result:** WARN +``` +kubectl --kubeconfig=/root/.kube/config get psp -o json | jq .items[] | jq -r 'select((.spec.allowPrivilegeEscalation == null) or (.spec.allowPrivilegeEscalation == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' +``` -**Remediation:** -Create a PSP as described in the Kubernetes documentation, ensuring that the -`.spec.runAsUser.rule` is set to either `MustRunAsNonRoot` or `MustRunAs` with the range of -UIDs not including `0`. +**Expected result**: -#### 5.2.7 Minimize the admission of containers with the `NET_RAW` capability (Not Scored) - -**Result:** WARN - -**Remediation:** -Create a PSP as described in the Kubernetes documentation, ensuring that the -`.spec.requiredDropCapabilities` is set to include either `NET_RAW` or `ALL`. - -#### 5.2.8 Minimize the admission of containers with added capabilities (Not Scored) - -**Result:** WARN - -**Remediation:** -Ensure that `allowedCapabilities` is not present in PSPs for the cluster unless -it is set to an empty array. - -#### 5.2.9 Minimize the admission of containers with capabilities assigned (Not Scored) - -**Result:** WARN - -**Remediation:** -Review the use of capabilites in applications runnning on your cluster. Where a namespace -contains applicaions which do not require any Linux capabities to operate consider adding -a PSP which forbids the admission of containers which do not drop all capabilities. +``` +1 is greater than 0 +``` ### 5.3 Network Policies and CNI -#### 5.3.1 Ensure that the CNI in use supports Network Policies (Not Scored) - -**Result:** WARN - -**Remediation:** -If the CNI plugin in use does not support network policies, consideration should be given to -making use of a different plugin, or finding an alternate mechanism for restricting traffic -in the Kubernetes cluster. - #### 5.3.2 Ensure that all Namespaces have Network Policies defined (Scored) -**Result:** WARN +**Result:** PASS **Remediation:** Follow the documentation and create `NetworkPolicy` objects as you need them. -### 5.4 Secrets Management +**Audit:** -#### 5.4.1 Prefer using secrets as files over secrets as environment variables (Not Scored) +``` +/mnt/kube-bench/test_helpers/5.3.2.sh +``` -**Result:** WARN +**Expected result**: -**Remediation:** -if possible, rewrite application code to read secrets from mounted secret files, rather than -from environment variables. - -#### 5.4.2 Consider external secret storage (Not Scored) - -**Result:** WARN - -**Remediation:** -Refer to the secrets management options offered by your cloud provider or a third-party -secrets management solution. - -### 5.5 Extensible Admission Control - -#### 5.5.1 Configure Image Provenance using `ImagePolicyWebhook` admission controller (Not Scored) - -**Result:** WARN - -**Remediation:** -Follow the Kubernetes documentation and setup image provenance. +``` +'--pass' is present +``` ### 5.6 General Policies -#### 5.6.1 Create administrative boundaries between resources using namespaces (Not Scored) - -**Result:** WARN - -**Remediation:** -Follow the documentation and create namespaces for objects in your deployment as you need -them. - -#### 5.6.2 Ensure that the seccomp profile is set to docker/default in your pod definitions (Not Scored) - -**Result:** WARN - -**Remediation:** -Seccomp is an alpha feature currently. By default, all alpha features are disabled. So, you -would need to enable alpha features in the apiserver by passing `"--feature- -gates=AllAlpha=true"` argument. -Edit the `/etc/kubernetes/apiserver` file on the master node and set the `KUBE_API_ARGS` -parameter to `"--feature-gates=AllAlpha=true"` -`KUBE_API_ARGS="--feature-gates=AllAlpha=true"` -Based on your system, restart the kube-apiserver service. For example: - -``` bash -systemctl restart kube-apiserver.service -``` - -Use annotations to enable the docker/default seccomp profile in your pod definitions. An -example is as below: - -``` bash -apiVersion: v1 -kind: Pod -metadata: - name: trustworthy-pod - annotations: - seccomp.security.alpha.kubernetes.io/pod: docker/default -spec: - containers: - - name: trustworthy-container - image: sotrustworthy:latest -``` - -#### 5.6.3 Apply Security Context to Your Pods and Containers (Not Scored) - -**Result:** WARN - -**Remediation:** -Follow the Kubernetes documentation and apply security contexts to your pods. For a -suggested list of security contexts, you may refer to the CIS Security Benchmark for Docker -Containers. - #### 5.6.4 The default namespace should not be used (Scored) -**Result:** WARN +**Result:** PASS **Remediation:** Ensure that namespaces are created to allow for appropriate segregation of Kubernetes resources and that all new resources are created in a specific namespace. +**Audit:** + +``` +/mnt/kube-bench/test_helpers/5.6.4.sh +``` + +**Expected result**: + +``` +'0' is equal to '0' +``` + diff --git a/content/rancher/v2.x/en/security/hardening-2.3.4/_index.md b/content/rancher/v2.x/en/security/hardening-2.3.4/_index.md deleted file mode 100644 index 6b437ce37f3..00000000000 --- a/content/rancher/v2.x/en/security/hardening-2.3.4/_index.md +++ /dev/null @@ -1,491 +0,0 @@ ---- -title: Hardening Guide v2.3.4 -weight: 100 ---- - -This document provides prescriptive guidance for hardening a production installation of Rancher v2.3.4. It outlines the configurations and controls required to address Kubernetes benchmark controls from the Center for Information Security (CIS). - -> This hardening guide describes how to secure the nodes in your cluster, and it is recommended to follow this guide before installing Kubernetes. - -This hardening guide is intended to be used with specific versions of the CIS Kubernetes Benchmark, Kubernetes, and Rancher: - -Hardening Guide Version | Rancher Version | CIS Benchmark Version | Kubernetes Version -------------------------|----------------|-----------------------|------------------ -Hardening Guide v2.3.4 | Rancher v2.3.4 | Benchmark v1.5 | Kubernetes 1.15 - - -[Click here to download a PDF version of this document](https://releases.rancher.com/documents/security/2.3.4/Rancher_Hardening_Guide.pdf) - -### Overview - -This document provides prescriptive guidance for hardening a production installation of Rancher v2.3.4 with Kubernetes v1.15. It outlines the configurations required to address Kubernetes benchmark controls from the Center for Information Security (CIS). - -For more detail about evaluating a hardened cluster against the official CIS benchmark, refer to the [CIS Benchmark Rancher Self-Assessment Guide - Rancher v2.3.4]({{< baseurl >}}/rancher/v2.x/en/security/benchmark-2.3.4/). - -### Configure Kernel Runtime Parameters - -The folowing `sysctl` configuration is recommended for all nodes type in the cluster. Set the following parameters in `/etc/sysctl.d/90-kubelet.conf`: - -``` bash -vm.overcommit_memory=1 -vm.panic_on_oom=0 -kernel.panic=10 -kernel.panic_on_oops=1 -kernel.keys.root_maxkeys=1000000 -kernel.keys.root_maxbytes=25000000 -``` - -Run `sysctl -p /etc/sysctl.d/90-kubelet.conf` to enable the settings. - -### Configuration Files and Permissions. - -#### kubelet.conf - -**path**: /etc/sysctl.d/kubelet.conf - -**owner**: root:root - -**permissions:** 0644 - -**contents**: - -``` text -vm.overcommit_memory=1 -kernel.panic=10 -kernel.panic_on_oops=1 -``` - -#### admission.yaml - -**path**: /opt/kubernetes/admission.yaml - -**owner**: root:root - -**permissions**: 0600 - -**content**: - -``` yaml -apiVersion: apiserver.k8s.io/v1alpha1 -kind: AdmissionConfiguration -plugins: -- name: EventRateLimit - path: /opt/kubernetes/event.yaml -``` - -#### event.yaml - -**path**: /opt/kubernetes/event.yaml - -**owner**: root:root - -**permissions**: 0600 - -**content**: - -``` yaml -apiVersion: eventratelimit.admission.k8s.io/v1alpha1 -kind: Configuration -limits: -- type: Server - qps: 5000 - burst: 20000 -``` - -#### encryption.yaml - -**path**: /opt/kubernetes/encryption.yaml - -**owner**: root:root - -**permissions**: 0600 - -**content**: - -``` yaml -apiVersion: apiserver.config.k8s.io/v1 -kind: EncryptionConfiguration -resources: - - resources: - - secrets - providers: - - aescbc: - keys: - - name: key1 - secret: - - identity: {} -``` - - -#### audit.yaml - -**path**: /opt/kubernetes/audit.yaml - -**owner**: root:root - -**permissions**: 0600 - -**content**: - -``` yaml -apiVersion: audit.k8s.io/v1beta1 -kind: Policy -rules: -- level: Metadata - -``` - -### Configure `etcd` user and `data-dir` permissions - -#### create `etcd` user and `group` - -``` -addgroup --gid 52034 etcd -useradd --comment "etcd service account" --uid 52034 --gid 52034 etcd -``` - -#### create `data-dir` and set permissions -``` -mkdir -p /var/lib/etcd && chown etcd.etcd /var/lib/etcd && chmod 0700 /var/lib/etcd -``` - - -### Hardened RKE `config.yml` configuration - -``` yaml -# -# Cluster Config -# -docker_root_dir: /var/lib/docker -enable_cluster_alerting: false -enable_cluster_monitoring: false -enable_network_policy: false -# -# Rancher Config -# -rancher_kubernetes_engine_config: - addon_job_timeout: 30 - addons: |- - --- - apiVersion: v1 - kind: Namespace - metadata: - name: ingress-nginx - --- - apiVersion: rbac.authorization.k8s.io/v1 - kind: Role - metadata: - name: default-psp-role - namespace: ingress-nginx - rules: - - apiGroups: - - extensions - resourceNames: - - default-psp - resources: - - podsecuritypolicies - verbs: - - use - --- - apiVersion: rbac.authorization.k8s.io/v1 - kind: RoleBinding - metadata: - name: default-psp-rolebinding - namespace: ingress-nginx - roleRef: - apiGroup: rbac.authorization.k8s.io - kind: Role - name: default-psp-role - subjects: - - apiGroup: rbac.authorization.k8s.io - kind: Group - name: system:serviceaccounts - - apiGroup: rbac.authorization.k8s.io - kind: Group - name: system:authenticated - --- - apiVersion: v1 - kind: Namespace - metadata: - name: cattle-system - --- - apiVersion: rbac.authorization.k8s.io/v1 - kind: Role - metadata: - name: default-psp-role - namespace: cattle-system - rules: - - apiGroups: - - extensions - resourceNames: - - default-psp - resources: - - podsecuritypolicies - verbs: - - use - --- - apiVersion: rbac.authorization.k8s.io/v1 - kind: RoleBinding - metadata: - name: default-psp-rolebinding - namespace: cattle-system - roleRef: - apiGroup: rbac.authorization.k8s.io - kind: Role - name: default-psp-role - subjects: - - apiGroup: rbac.authorization.k8s.io - kind: Group - name: system:serviceaccounts - - apiGroup: rbac.authorization.k8s.io - kind: Group - name: system:authenticated - --- - apiVersion: extensions/v1beta1 - kind: PodSecurityPolicy - metadata: - name: restricted - spec: - requiredDropCapabilities: - - NET_RAW - privileged: false - allowPrivilegeEscalation: false - defaultAllowPrivilegeEscalation: false - fsGroup: - rule: RunAsAny - runAsUser: - rule: MustRunAsNonRoot - seLinux: - rule: RunAsAny - supplementalGroups: - rule: RunAsAny - volumes: - - emptyDir - - secret - - persistentVolumeClaim - - downwardAPI - - configMap - - projected - --- - apiVersion: rbac.authorization.k8s.io/v1 - kind: ClusterRole - metadata: - name: psp:restricted - rules: - - apiGroups: - - extensions - resourceNames: - - restricted - resources: - - podsecuritypolicies - verbs: - - use - --- - apiVersion: rbac.authorization.k8s.io/v1 - kind: ClusterRoleBinding - metadata: - name: psp:restricted - roleRef: - apiGroup: rbac.authorization.k8s.io - kind: ClusterRole - name: psp:restricted - subjects: - - apiGroup: rbac.authorization.k8s.io - kind: Group - name: system:serviceaccounts - - apiGroup: rbac.authorization.k8s.io - kind: Group - name: system:authenticated - --- - apiVersion: v1 - kind: ServiceAccount - metadata: - name: tiller - namespace: kube-system - --- - apiVersion: rbac.authorization.k8s.io/v1 - kind: ClusterRoleBinding - metadata: - name: tiller - roleRef: - apiGroup: rbac.authorization.k8s.io - kind: ClusterRole - name: cluster-admin - subjects: - - kind: ServiceAccount - name: tiller - namespace: kube-system - ignore_docker_version: true - kubernetes_version: v1.15.6-rancher1-2 -# -# If you are using calico on AWS -# -# network: -# plugin: calico -# calico_network_provider: -# cloud_provider: aws -# -# # To specify flannel interface -# -# network: -# plugin: flannel -# flannel_network_provider: -# iface: eth1 -# -# # To specify flannel interface for canal plugin -# -# network: -# plugin: canal -# canal_network_provider: -# iface: eth1 -# - network: - mtu: 0 - plugin: canal -# -# services: -# kube-api: -# service_cluster_ip_range: 10.43.0.0/16 -# kube-controller: -# cluster_cidr: 10.42.0.0/16 -# service_cluster_ip_range: 10.43.0.0/16 -# kubelet: -# cluster_domain: cluster.local -# cluster_dns_server: 10.43.0.10 -# - services: - etcd: - backup_config: - enabled: false - interval_hours: 12 - retention: 6 - safe_timestamp: false - creation: 12h - extra_args: - data-dir: /var/lib/etcd - extra_binds: - - '/var/lib/etcd:/var/lib/etcd' - gid: 52034 - retention: 72h - snapshot: false - uid: 52034 - kube_api: - always_pull_images: false - extra_args: - admission-control-config-file: /opt/kubernetes/admission.yaml - anonymous-auth: 'false' - audit-log-format: json - audit-log-maxage: '30' - audit-log-maxbackup: '10' - audit-log-maxsize: '100' - audit-log-path: /var/log/kube-audit/audit-log.json - audit-policy-file: /opt/kubernetes/audit.yaml - enable-admission-plugins: >- - ServiceAccount,NamespaceLifecycle,LimitRanger,PersistentVolumeLabel,DefaultStorageClass,ResourceQuota,DefaultTolerationSeconds,AlwaysPullImages,DenyEscalatingExec,NodeRestriction,EventRateLimit,PodSecurityPolicy - encryption-provider-config: /opt/kubernetes/encryption.yaml - profiling: 'false' - service-account-lookup: 'true' - tls-cipher-suites: >- - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 - extra_binds: - - '/var/log/kube-audit:/var/log/kube-audit' - - '/opt/kubernetes:/opt/kubernetes' - pod_security_policy: true - service_node_port_range: 30000-32767 - kube_controller: - extra_args: - address: 127.0.0.1 - feature-gates: RotateKubeletServerCertificate=true - profiling: 'false' - terminated-pod-gc-threshold: '1000' - kubelet: - extra_args: - anonymous-auth: 'false' - event-qps: '0' - feature-gates: RotateKubeletServerCertificate=true - make-iptables-util-chains: 'true' - protect-kernel-defaults: 'true' - streaming-connection-idle-timeout: 1800s - tls-cipher-suites: >- - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 - fail_swap_on: false - generate_serving_certificate: true - scheduler: - extra_args: - address: 127.0.0.1 - profiling: 'false' - ssh_agent_auth: false -windows_prefered_cluster: false -``` - -### Hardened Example Ubuntu cloud-config: - -``` yaml -#cloud-config -package_update: false -packages: - - curl - - jq -runcmd: - - sysctl -w vm.overcommit_memory=1 - - sysctl -w kernel.panic=10 - - sysctl -w kernel.panic_on_oops=1 - - curl https://releases.rancher.com/install-docker/18.09.sh | sh - - usermod -aG docker ubuntu - - return=1; while [ $return != 0 ]; do sleep 2; docker ps; return=$?; done - - addgroup --gid 52034 etcd - - useradd --comment "etcd service account" --uid 52034 --gid 52034 etcd - - mkdir -p /var/lib/etcd && chown etcd.etcd /var/lib/etcd && chmod 0700 /var/lib/etcd - - ${agent_cmd} --etcd --controlplane --worker - - mkdir /mnt/kube-bench -write_files: - - path: /etc/sysctl.d/kubelet.conf - owner: root:root - permissions: "0644" - content: | - vm.overcommit_memory=1 - kernel.panic=10 - kernel.panic_on_oops=1 - - path: /opt/kubernetes/admission.yaml - owner: root:root - permissions: "0600" - content: | - apiVersion: apiserver.k8s.io/v1alpha1 - kind: AdmissionConfiguration - plugins: - - name: EventRateLimit - path: /opt/kubernetes/event.yaml - - path: /opt/kubernetes/event.yaml - owner: root:root - permissions: "0600" - content: | - apiVersion: eventratelimit.admission.k8s.io/v1alpha1 - kind: Configuration - limits: - - type: Server - qps: 5000 - burst: 20000 - - path: /opt/kubernetes/encryption.yaml - owner: root:root - permissions: "0600" - content: | - apiVersion: apiserver.config.k8s.io/v1 - kind: EncryptionConfiguration - resources: - - resources: - - secrets - providers: - - aescbc: - keys: - - name: key1 - secret: LF7YiCFyWqAa2MovOgp42rArBdLBGWdjJpX2knvYAkc= - - identity: {} - - path: /opt/kubernetes/audit.yaml - owner: root:root - permissions: "0600" - content: | - apiVersion: audit.k8s.io/v1beta1 - kind: Policy - rules: - - level: Metadata -``` diff --git a/content/rancher/v2.x/en/security/hardening-2.3.5/_index.md b/content/rancher/v2.x/en/security/hardening-2.3.5/_index.md new file mode 100644 index 00000000000..80537a139d9 --- /dev/null +++ b/content/rancher/v2.x/en/security/hardening-2.3.5/_index.md @@ -0,0 +1,424 @@ +--- +title: Hardening Guide v2.3.5 +weight: 100 +--- + +This document provides prescriptive guidance for hardening a production installation of Rancher v2.3.5. It outlines the configurations and controls required to address Kubernetes benchmark controls from the Center for Information Security (CIS). + +> This hardening guide describes how to secure the nodes in your cluster, and it is recommended to follow this guide before installing Kubernetes. + +This hardening guide is intended to be used with specific versions of the CIS Kubernetes Benchmark, Kubernetes, and Rancher: + +Hardening Guide Version | Rancher Version | CIS Benchmark Version | Kubernetes Version +------------------------|----------------|-----------------------|------------------ +Hardening Guide v2.3.5 | Rancher v2.3.5 | Benchmark v1.5 | Kubernetes 1.15 + + +[Click here to download a PDF version of this document](https://releases.rancher.com/documents/security/2.3.5/Rancher_Hardening_Guide.pdf) + +### Overview + +This document provides prescriptive guidance for hardening a production installation of Rancher v2.3.5 with Kubernetes v1.15. It outlines the configurations required to address Kubernetes benchmark controls from the Center for Information Security (CIS). + +For more detail about evaluating a hardened cluster against the official CIS benchmark, refer to the [CIS Benchmark Rancher Self-Assessment Guide - Rancher v2.3.5]({{< baseurl >}}/rancher/v2.x/en/security/benchmark-2.3.5/). + +### Configure Kernel Runtime Parameters + +The folowing `sysctl` configuration is recommended for all nodes type in the cluster. Set the following parameters in `/etc/sysctl.d/90-kubelet.conf`: + +``` bash +vm.overcommit_memory=1 +vm.panic_on_oom=0 +kernel.panic=10 +kernel.panic_on_oops=1 +kernel.keys.root_maxbytes=25000000 +``` + +Run `sysctl -p /etc/sysctl.d/90-kubelet.conf` to enable the settings. + +### Configure `etcd` user and group +A user account and group for the **etcd** service is required to be setup prior to installing RKE. The **uid** and **gid** for the **etcd** user will be used in the RKE **config.yml** to set the proper permissions for files and directories during installation time. + +#### create `etcd` user and group +To create the **etcd** group run the following console commands. + +``` +addgroup --gid 52034 etcd +useradd --comment "etcd service account" --uid 52034 --gid 52034 etcd +``` + +Update the RKE **config.yml** with the **uid** and **gid** of the **etcd** user: + +``` yaml +services: + etcd: + gid: 52034 + uid: 52034 +``` + +#### Set `automountServiceAccountToken` to `false` for `default` service accounts +Kubernetes provides a default service account which is used by cluster workloads where no specific service account is assigned to the pod. Where access to the Kubernetes API from a pod is required, a specific service account should be created for that pod, and rights granted to that service account. The default service account should be configured such that it does not provide a service account token and does not have any explicit rights assignments. + +For each namespace the **default** service account must include this value: + +``` +automountServiceAccountToken: false +``` + +Save the following yaml to a file called `account_update.yaml` + +``` yaml +apiVersion: v1 +kind: ServiceAccount +metadata: + name: default +automountServiceAccountToken: false +``` + +Create a bash script file called `account_update.sh`. Be sure to `chmod +x account_update.sh` so the script has execute permissions. + +``` +#!/bin/bash -e + +for namespace in $(kubectl get namespaces -A -o json | jq -r '.items[].metadata.name'); do + kubectl patch serviceaccount default -n ${namespace} -p "$(cat account_update.yaml)" +done +``` + +### Ensure that all Namespaces have Network Policies defined + +Running different applications on the same Kubernetes cluster creates a risk of one +compromised application attacking a neighboring application. Network segmentation is +important to ensure that containers can communicate only with those they are supposed +to. A network policy is a specification of how selections of pods are allowed to +communicate with each other and other network endpoints. + +Network Policies are namespace scoped. When a network policy is introduced to a given +namespace, all traffic not allowed by the policy is denied. However, if there are no network +policies in a namespace all traffic will be allowed into and out of the pods in that +namespace. + +> todo: add information about network policies and provide default example here: + + + +### Reference Hardened RKE `config.yml` configuration + +``` yaml +# If you intened to deploy Kubernetes in an air-gapped environment, +# please consult the documentation on how to configure custom RKE images. +kubernetes_version: "v1.15.9-rancher1-1" +enable_network_policy: true +default_pod_security_policy_template_id: "restricted" +nodes: +- address: "172.16.16.9" + port: "" + internal_address: "" + role: + - controlplane + - etcd + - worker + hostname_override: "" + user: "ubuntu" +services: + etcd: + uid: 52034 + gid: 52034 + kube-api: + pod_security_policy: true + secrets_encryption_config: + enabled: true + audit_log: + enabled: true + admission_configuration: + event_rate_limit: + enabled: true + kube-controller: + extra_args: + feature-gates: "RotateKubeletServerCertificate=true" + scheduler: + image: "" + extra_args: {} + extra_binds: [] + extra_env: [] + kubelet: + generate_serving_certificate: true + extra_args: + feature-gates: "RotateKubeletServerCertificate=true" + protect-kernel-defaults: "true" + tls-cipher-suites: "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256" + extra_binds: [] + extra_env: [] + cluster_domain: "" + infra_container_image: "" + cluster_dns_server: "" + fail_swap_on: false + generate_serving_certificate: true + kubeproxy: + image: "" + extra_args: {} + extra_binds: [] + extra_env: [] +network: + plugin: "" + options: {} + mtu: 0 + node_selector: {} +authentication: + strategy: "" + sans: [] + webhook: null +addons: | + --- + apiVersion: v1 + kind: Namespace + metadata: + name: ingress-nginx + --- + apiVersion: rbac.authorization.k8s.io/v1 + kind: Role + metadata: + name: default-psp-role + namespace: ingress-nginx + rules: + - apiGroups: + - extensions + resourceNames: + - default-psp + resources: + - podsecuritypolicies + verbs: + - use + --- + apiVersion: rbac.authorization.k8s.io/v1 + kind: RoleBinding + metadata: + name: default-psp-rolebinding + namespace: ingress-nginx + roleRef: + apiGroup: rbac.authorization.k8s.io + kind: Role + name: default-psp-role + subjects: + - apiGroup: rbac.authorization.k8s.io + kind: Group + name: system:serviceaccounts + - apiGroup: rbac.authorization.k8s.io + kind: Group + name: system:authenticated + --- + apiVersion: v1 + kind: Namespace + metadata: + name: cattle-system + --- + apiVersion: rbac.authorization.k8s.io/v1 + kind: Role + metadata: + name: default-psp-role + namespace: cattle-system + rules: + - apiGroups: + - extensions + resourceNames: + - default-psp + resources: + - podsecuritypolicies + verbs: + - use + --- + apiVersion: rbac.authorization.k8s.io/v1 + kind: RoleBinding + metadata: + name: default-psp-rolebinding + namespace: cattle-system + roleRef: + apiGroup: rbac.authorization.k8s.io + kind: Role + name: default-psp-role + subjects: + - apiGroup: rbac.authorization.k8s.io + kind: Group + name: system:serviceaccounts + - apiGroup: rbac.authorization.k8s.io + kind: Group + name: system:authenticated + --- + apiVersion: extensions/v1beta1 + kind: PodSecurityPolicy + metadata: + name: restricted + spec: + requiredDropCapabilities: + - NET_RAW + privileged: false + allowPrivilegeEscalation: false + defaultAllowPrivilegeEscalation: false + fsGroup: + rule: RunAsAny + runAsUser: + rule: MustRunAsNonRoot + seLinux: + rule: RunAsAny + supplementalGroups: + rule: RunAsAny + volumes: + - emptyDir + - secret + - persistentVolumeClaim + - downwardAPI + - configMap + - projected + --- + apiVersion: rbac.authorization.k8s.io/v1 + kind: ClusterRole + metadata: + name: psp:restricted + rules: + - apiGroups: + - extensions + resourceNames: + - restricted + resources: + - podsecuritypolicies + verbs: + - use + --- + apiVersion: rbac.authorization.k8s.io/v1 + kind: ClusterRoleBinding + metadata: + name: psp:restricted + roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: psp:restricted + subjects: + - apiGroup: rbac.authorization.k8s.io + kind: Group + name: system:serviceaccounts + - apiGroup: rbac.authorization.k8s.io + kind: Group + name: system:authenticated + --- + apiVersion: v1 + kind: ServiceAccount + metadata: + name: tiller + namespace: kube-system + --- + apiVersion: rbac.authorization.k8s.io/v1 + kind: ClusterRoleBinding + metadata: + name: tiller + roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: cluster-admin + subjects: + - kind: ServiceAccount + name: tiller + namespace: kube-system + +addons_include: [] +system_images: + etcd: "" + alpine: "" + nginx_proxy: "" + cert_downloader: "" + kubernetes_services_sidecar: "" + kubedns: "" + dnsmasq: "" + kubedns_sidecar: "" + kubedns_autoscaler: "" + coredns: "" + coredns_autoscaler: "" + kubernetes: "" + flannel: "" + flannel_cni: "" + calico_node: "" + calico_cni: "" + calico_controllers: "" + calico_ctl: "" + calico_flexvol: "" + canal_node: "" + canal_cni: "" + canal_flannel: "" + canal_flexvol: "" + weave_node: "" + weave_cni: "" + pod_infra_container: "" + ingress: "" + ingress_backend: "" + metrics_server: "" + windows_pod_infra_container: "" +ssh_key_path: "" +ssh_cert_path: "" +ssh_agent_auth: false +authorization: + mode: "" + options: {} +ignore_docker_version: false +private_registries: [] +ingress: + provider: "" + options: {} + node_selector: {} + extra_args: {} + dns_policy: "" + extra_envs: [] + extra_volumes: [] + extra_volume_mounts: [] +cluster_name: "" +prefix_path: "" +addon_job_timeout: 0 +bastion_host: + address: "" + port: "" + user: "" + ssh_key: "" + ssh_key_path: "" + ssh_cert: "" + ssh_cert_path: "" +monitoring: + provider: "" + options: {} + node_selector: {} +restore: + restore: false + snapshot_name: "" +dns: null +``` + +### Reference Hardened RKE Template configuration + +``` yaml +todo: + +``` + + +### Hardened Reference Ubuntu **cloud-config**: + +``` yaml +#cloud-config +packages: + - curl + - jq +runcmd: + - sysctl -w vm.overcommit_memory=1 + - sysctl -w kernel.panic=10 + - sysctl -w kernel.panic_on_oops=1 + - curl https://releases.rancher.com/install-docker/18.09.sh | sh + - usermod -aG docker ubuntu + - return=1; while [ $return != 0 ]; do sleep 2; docker ps; return=$?; done + - addgroup --gid 52034 etcd + - useradd --comment "etcd service account" --uid 52034 --gid 52034 etcd +write_files: + - path: /etc/sysctl.d/kubelet.conf + owner: root:root + permissions: "0644" + content: | + vm.overcommit_memory=1 + kernel.panic=10 + kernel.panic_on_oops=1 +``` From 6ccad619fb94776b1d3ad96385db7a92bca0b1a5 Mon Sep 17 00:00:00 2001 From: Nelson Roberts Date: Thu, 13 Feb 2020 13:23:19 -0700 Subject: [PATCH 06/14] EIO-247: added network policy section --- .../en/security/hardening-2.3.5/_index.md | 38 +++++++++++++++++-- 1 file changed, 35 insertions(+), 3 deletions(-) diff --git a/content/rancher/v2.x/en/security/hardening-2.3.5/_index.md b/content/rancher/v2.x/en/security/hardening-2.3.5/_index.md index 80537a139d9..e1b9e70715d 100644 --- a/content/rancher/v2.x/en/security/hardening-2.3.5/_index.md +++ b/content/rancher/v2.x/en/security/hardening-2.3.5/_index.md @@ -26,7 +26,7 @@ For more detail about evaluating a hardened cluster against the official CIS ben The folowing `sysctl` configuration is recommended for all nodes type in the cluster. Set the following parameters in `/etc/sysctl.d/90-kubelet.conf`: -``` bash +``` vm.overcommit_memory=1 vm.panic_on_oom=0 kernel.panic=10 @@ -96,11 +96,43 @@ communicate with each other and other network endpoints. Network Policies are namespace scoped. When a network policy is introduced to a given namespace, all traffic not allowed by the policy is denied. However, if there are no network policies in a namespace all traffic will be allowed into and out of the pods in that -namespace. +namespace. To use network policies, you must be using a networking solution which supports `NetworkPolicy`. +A CNI (container network interface) plugin can provide the needed `NetworkPolicy` resource. +For this guide [canal](https://github.com/projectcalico/canal) will be used to provide +the `NetworkPolicy` resource. Additional information about CNI providers can be found +[here](https://rancher.com/blog/2019/2019-03-21-comparing-kubernetes-cni-providers-flannel-calico-canal-and-weave/) -> todo: add information about network policies and provide default example here: +Once a CNI provider is enabled on a cluster that supports the `NetworkPolicy` resouce a default network policy +can be applied. For reference purposes a **permissive** example is provide below. If you want to +allow all traffic to all pods in a namespace (even if policies are added that cause some pods to be treated as “isolated”), +you can create a policy that explicitly allows all traffic in that namespace. Save the following `yaml` as +`default-allow-all.yaml` +``` yaml +--- +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: default-allow-all +spec: + podSelector: {} + policyTypes: + - Ingress + - Egress +``` + +Create a bash script file called `apply_networkPolicy_to_all_ns.sh`. Be sure to +`chmod +x apply_networkPolicy_to_all_ns.sh` so the script has execute permissions. + +``` +#!/bin/bash -e + +for namespace in $(kubectl get namespaces -A -o json | jq -r '.items[].metadata.name'); do + kubectl apply -f default-allow-all.yaml -n ${namespace} +done +``` +Execute this scipt to apply the `default-allow-all.yaml` the **permissive** `NetworkPolicy` to all namespaces. ### Reference Hardened RKE `config.yml` configuration From fa225cf6654780e6bcc7ea2f9d3066377a606a74 Mon Sep 17 00:00:00 2001 From: Nelson Roberts Date: Thu, 13 Feb 2020 14:22:10 -0700 Subject: [PATCH 07/14] EIO-247: spelling correction; update RKE config, Template and cloud-config sections --- .../en/security/hardening-2.3.5/_index.md | 278 +++++++++++++++++- 1 file changed, 263 insertions(+), 15 deletions(-) diff --git a/content/rancher/v2.x/en/security/hardening-2.3.5/_index.md b/content/rancher/v2.x/en/security/hardening-2.3.5/_index.md index e1b9e70715d..594810b2114 100644 --- a/content/rancher/v2.x/en/security/hardening-2.3.5/_index.md +++ b/content/rancher/v2.x/en/security/hardening-2.3.5/_index.md @@ -132,9 +132,12 @@ for namespace in $(kubectl get namespaces -A -o json | jq -r '.items[].metadata. kubectl apply -f default-allow-all.yaml -n ${namespace} done ``` -Execute this scipt to apply the `default-allow-all.yaml` the **permissive** `NetworkPolicy` to all namespaces. +Execute this script to apply the `default-allow-all.yaml` the **permissive** `NetworkPolicy` to all namespaces. -### Reference Hardened RKE `config.yml` configuration +### Reference Hardened RKE `cluster.yml` configuration +The reference `cluster.yml` provides the configuration needed to acheive a hardened install +of Rancher Kubernetes Engine (RKE). Install [documentation](https://rancher.com/docs/rke/latest/en/installation/) is +provided with additional details about the configuration items. ``` yaml # If you intened to deploy Kubernetes in an air-gapped environment, @@ -142,16 +145,6 @@ Execute this scipt to apply the `default-allow-all.yaml` the **permissive** `Net kubernetes_version: "v1.15.9-rancher1-1" enable_network_policy: true default_pod_security_policy_template_id: "restricted" -nodes: -- address: "172.16.16.9" - port: "" - internal_address: "" - role: - - controlplane - - etcd - - worker - hostname_override: "" - user: "ubuntu" services: etcd: uid: 52034 @@ -423,14 +416,269 @@ dns: null ### Reference Hardened RKE Template configuration -``` yaml -todo: +The reference RKE Template provides the configuration needed to acheive a hardened install of Kubenetes. +RKE Templates are used to provision Kubernetes and define Rancher settings. Follow the Rancher +[documentaion](https://rancher.com/docs/rancher/v2.x/en/installation) for additional installation and RKE Template details. +``` yaml +# +# Cluster Config +# +default_pod_security_policy_template_id: restricted +docker_root_dir: /var/lib/docker +enable_cluster_alerting: false +enable_cluster_monitoring: false +enable_network_policy: true +# +# Rancher Config +# +rancher_kubernetes_engine_config: + addon_job_timeout: 30 + addons: |- + --- + apiVersion: v1 + kind: Namespace + metadata: + name: ingress-nginx + --- + apiVersion: rbac.authorization.k8s.io/v1 + kind: Role + metadata: + name: default-psp-role + namespace: ingress-nginx + rules: + - apiGroups: + - extensions + resourceNames: + - default-psp + resources: + - podsecuritypolicies + verbs: + - use + --- + apiVersion: rbac.authorization.k8s.io/v1 + kind: RoleBinding + metadata: + name: default-psp-rolebinding + namespace: ingress-nginx + roleRef: + apiGroup: rbac.authorization.k8s.io + kind: Role + name: default-psp-role + subjects: + - apiGroup: rbac.authorization.k8s.io + kind: Group + name: system:serviceaccounts + - apiGroup: rbac.authorization.k8s.io + kind: Group + name: system:authenticated + --- + apiVersion: v1 + kind: Namespace + metadata: + name: cattle-system + --- + apiVersion: rbac.authorization.k8s.io/v1 + kind: Role + metadata: + name: default-psp-role + namespace: cattle-system + rules: + - apiGroups: + - extensions + resourceNames: + - default-psp + resources: + - podsecuritypolicies + verbs: + - use + --- + apiVersion: rbac.authorization.k8s.io/v1 + kind: RoleBinding + metadata: + name: default-psp-rolebinding + namespace: cattle-system + roleRef: + apiGroup: rbac.authorization.k8s.io + kind: Role + name: default-psp-role + subjects: + - apiGroup: rbac.authorization.k8s.io + kind: Group + name: system:serviceaccounts + - apiGroup: rbac.authorization.k8s.io + kind: Group + name: system:authenticated + --- + apiVersion: extensions/v1beta1 + kind: PodSecurityPolicy + metadata: + name: restricted + spec: + requiredDropCapabilities: + - NET_RAW + privileged: false + allowPrivilegeEscalation: false + defaultAllowPrivilegeEscalation: false + fsGroup: + rule: RunAsAny + runAsUser: + rule: MustRunAsNonRoot + seLinux: + rule: RunAsAny + supplementalGroups: + rule: RunAsAny + volumes: + - emptyDir + - secret + - persistentVolumeClaim + - downwardAPI + - configMap + - projected + --- + apiVersion: rbac.authorization.k8s.io/v1 + kind: ClusterRole + metadata: + name: psp:restricted + rules: + - apiGroups: + - extensions + resourceNames: + - restricted + resources: + - podsecuritypolicies + verbs: + - use + --- + apiVersion: rbac.authorization.k8s.io/v1 + kind: ClusterRoleBinding + metadata: + name: psp:restricted + roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: psp:restricted + subjects: + - apiGroup: rbac.authorization.k8s.io + kind: Group + name: system:serviceaccounts + - apiGroup: rbac.authorization.k8s.io + kind: Group + name: system:authenticated + --- + apiVersion: v1 + kind: ServiceAccount + metadata: + name: tiller + namespace: kube-system + --- + apiVersion: rbac.authorization.k8s.io/v1 + kind: ClusterRoleBinding + metadata: + name: tiller + roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: cluster-admin + subjects: + - kind: ServiceAccount + name: tiller + namespace: kube-system + ignore_docker_version: true + kubernetes_version: v1.15.9-rancher1-1 +# +# If you are using calico on AWS +# +# network: +# plugin: calico +# calico_network_provider: +# cloud_provider: aws +# +# # To specify flannel interface +# +# network: +# plugin: flannel +# flannel_network_provider: +# iface: eth1 +# +# # To specify flannel interface for canal plugin +# +# network: +# plugin: canal +# canal_network_provider: +# iface: eth1 +# + network: + mtu: 0 + plugin: canal +# +# services: +# kube-api: +# service_cluster_ip_range: 10.43.0.0/16 +# kube-controller: +# cluster_cidr: 10.42.0.0/16 +# service_cluster_ip_range: 10.43.0.0/16 +# kubelet: +# cluster_domain: cluster.local +# cluster_dns_server: 10.43.0.10 +# + services: + etcd: + backup_config: + enabled: false + interval_hours: 12 + retention: 6 + safe_timestamp: false + creation: 12h + extra_args: + election-timeout: '5000' + heartbeat-interval: '500' + gid: 52034 + retention: 72h + snapshot: false + uid: 52034 + kube_api: + always_pull_images: false + audit_log: + enabled: true + event_rate_limit: + enabled: true + pod_security_policy: true + secrets_encryption_config: + enabled: true + service_node_port_range: 30000-32767 + kube_controller: + extra_args: + address: 127.0.0.1 + feature-gates: RotateKubeletServerCertificate=true + profiling: 'false' + terminated-pod-gc-threshold: '1000' + kubelet: + extra_args: + anonymous-auth: 'false' + event-qps: '0' + feature-gates: RotateKubeletServerCertificate=true + make-iptables-util-chains: 'true' + protect-kernel-defaults: 'true' + streaming-connection-idle-timeout: 1800s + tls-cipher-suites: >- + TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 + fail_swap_on: false + generate_serving_certificate: true + scheduler: + extra_args: + address: 127.0.0.1 + profiling: 'false' + ssh_agent_auth: false +windows_prefered_cluster: false ``` - ### Hardened Reference Ubuntu **cloud-config**: +The reference **cloud-config** is generenally used in cloud infrastructure environments to allow for +configuration managment of compute instances. The reference config configures Ubuntu operating system level settings +needed before installing kubernetes. + ``` yaml #cloud-config packages: From 3e6ae1ad0abe7f3bd5f351d14ffb4ca521b1e653 Mon Sep 17 00:00:00 2001 From: Nelson Roberts Date: Thu, 13 Feb 2020 15:39:42 -0700 Subject: [PATCH 08/14] EIO-247: clarify policy enforcement add link to k8s network policies site --- .../v2.x/en/security/hardening-2.3.5/_index.md | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/content/rancher/v2.x/en/security/hardening-2.3.5/_index.md b/content/rancher/v2.x/en/security/hardening-2.3.5/_index.md index 594810b2114..e8ee07d92de 100644 --- a/content/rancher/v2.x/en/security/hardening-2.3.5/_index.md +++ b/content/rancher/v2.x/en/security/hardening-2.3.5/_index.md @@ -96,18 +96,19 @@ communicate with each other and other network endpoints. Network Policies are namespace scoped. When a network policy is introduced to a given namespace, all traffic not allowed by the policy is denied. However, if there are no network policies in a namespace all traffic will be allowed into and out of the pods in that -namespace. To use network policies, you must be using a networking solution which supports `NetworkPolicy`. -A CNI (container network interface) plugin can provide the needed `NetworkPolicy` resource. -For this guide [canal](https://github.com/projectcalico/canal) will be used to provide -the `NetworkPolicy` resource. Additional information about CNI providers can be found +namespace. To enforce network policies, a CNI (container network interface) plugin must be enabled. +This guide uses [canal](https://github.com/projectcalico/canal) to provide the policy enforcement. +Additional information about CNI providers can be found [here](https://rancher.com/blog/2019/2019-03-21-comparing-kubernetes-cni-providers-flannel-calico-canal-and-weave/) -Once a CNI provider is enabled on a cluster that supports the `NetworkPolicy` resouce a default network policy -can be applied. For reference purposes a **permissive** example is provide below. If you want to -allow all traffic to all pods in a namespace (even if policies are added that cause some pods to be treated as “isolated”), +Once a CNI provider is enabled on a cluster a default network policy can be applied. For reference purposes a +**permissive** example is provide below. If you want to allow all traffic to all pods in a namespace +(even if policies are added that cause some pods to be treated as “isolated”), you can create a policy that explicitly allows all traffic in that namespace. Save the following `yaml` as -`default-allow-all.yaml` +`default-allow-all.yaml`. Additional [documentation](https://kubernetes.io/docs/concepts/services-networking/network-policies/) +about network poilices can be found on the Kubernetes site. +> This `NetworkPolicy` is not recommended for production use ``` yaml --- From e358ad9f2fb0a65fc843955782f20c5cf89fb5a4 Mon Sep 17 00:00:00 2001 From: Nelson Roberts Date: Thu, 13 Feb 2020 15:46:26 -0700 Subject: [PATCH 09/14] EIO-247: add refernce to RKE cli --- content/rancher/v2.x/en/security/hardening-2.3.5/_index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/rancher/v2.x/en/security/hardening-2.3.5/_index.md b/content/rancher/v2.x/en/security/hardening-2.3.5/_index.md index e8ee07d92de..fbf454ca480 100644 --- a/content/rancher/v2.x/en/security/hardening-2.3.5/_index.md +++ b/content/rancher/v2.x/en/security/hardening-2.3.5/_index.md @@ -136,7 +136,7 @@ done Execute this script to apply the `default-allow-all.yaml` the **permissive** `NetworkPolicy` to all namespaces. ### Reference Hardened RKE `cluster.yml` configuration -The reference `cluster.yml` provides the configuration needed to acheive a hardened install +The reference `cluster.yml` is used by the RKE cli that provides the configuration needed to acheive a hardened install of Rancher Kubernetes Engine (RKE). Install [documentation](https://rancher.com/docs/rke/latest/en/installation/) is provided with additional details about the configuration items. From 5e7ce073bd4a85ee211bcc01af1ade8838a41709 Mon Sep 17 00:00:00 2001 From: Nelson Roberts Date: Thu, 13 Feb 2020 16:28:51 -0700 Subject: [PATCH 10/14] EIO-247: add tool requirement; remove confusing statement update scoring note --- content/rancher/v2.x/en/security/benchmark-2.3.5/_index.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/content/rancher/v2.x/en/security/benchmark-2.3.5/_index.md b/content/rancher/v2.x/en/security/benchmark-2.3.5/_index.md index d06ce9b3af0..25960e9645a 100644 --- a/content/rancher/v2.x/en/security/benchmark-2.3.5/_index.md +++ b/content/rancher/v2.x/en/security/benchmark-2.3.5/_index.md @@ -27,9 +27,10 @@ For more detail about each audit, including rationales and remediations for fail Rancher and RKE install Kubernetes services via Docker containers. Configuration is defined by arguments passed to the container at the time of initialization, not via configuration files. -Scoring the commands is different in Rancher Labs than in the CIS Benchmark. Where the commands differ from the original CIS benchmark, the commands specific to Rancher Labs are provided for testing. Only **scored** tests will be covered in this guide. +Where control audits differ from the original CIS benchmark, the audit commands specific to Rancher Labs are provided for testing. +When performing the tests, you will need access to the Docker command line on the hosts of all three RKE roles. The commands also make use of the the [jq](https://stedolan.github.io/jq/) and [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/) (with valid config) tools to and are required in the testing and evaluation of test results. -When performing the tests, you will need access to the Docker command line on the hosts of all three RKE roles. The commands also make use of the the `jq` and `kubectl` (with valid config) commands to provide human-readable formatting. +> NOTE: only scored tests are covered in this guide. ### Controls From 0796234e6bbcbfdadb52a31a1dd3aaa24ff83862 Mon Sep 17 00:00:00 2001 From: Nelson Roberts Date: Fri, 14 Feb 2020 11:53:16 -0700 Subject: [PATCH 11/14] EIO-247: update PSP api; add test helper script contents --- .../en/security/benchmark-2.3.5/_index.md | 242 ++++++++++++++++-- .../en/security/hardening-2.3.5/_index.md | 2 +- 2 files changed, 225 insertions(+), 19 deletions(-) diff --git a/content/rancher/v2.x/en/security/benchmark-2.3.5/_index.md b/content/rancher/v2.x/en/security/benchmark-2.3.5/_index.md index 25960e9645a..1faeb172746 100644 --- a/content/rancher/v2.x/en/security/benchmark-2.3.5/_index.md +++ b/content/rancher/v2.x/en/security/benchmark-2.3.5/_index.md @@ -112,10 +112,22 @@ Run the below command (based on the etcd data directory found above). For exampl chmod 700 /var/lib/etcd ``` -**Audit:** +**Audit Script:** 1.1.11.sh ``` -/mnt/kube-bench/test_helpers/1.1.11.sh etcd +#!/bin/bash -e + +etcd_bin=${1} + +test_dir=$(ps -ef | grep ${etcd_bin} | grep -- --data-dir | sed 's%.*data-dir[= ]\([^ ]*\).*%\1%') + +docker inspect etcd | jq -r '.[].HostConfig.Binds[]' | grep "${test_dir}" | cut -d ":" -f 1 | xargs stat -c %a +``` + +**Audit Execution:** + +``` +./1.1.11.sh etcd ``` **Expected result**: @@ -142,10 +154,22 @@ For example, chown etcd:etcd /var/lib/etcd ``` -**Audit:** +**Audit Script:** 1.1.12.sh ``` -/mnt/kube-bench/test_helpers/1.1.12.sh etcd +#!/bin/bash -e + +etcd_bin=${1} + +test_dir=$(ps -ef | grep ${etcd_bin} | grep -- --data-dir | sed 's%.*data-dir[= ]\([^ ]*\).*%\1%') + +docker inspect etcd | jq -r '.[].HostConfig.Binds[]' | grep "${test_dir}" | cut -d ":" -f 1 | xargs stat -c %U:%G +``` + +**Audit Execution:** + +``` +./1.1.12.sh etcd ``` **Expected result**: @@ -234,10 +258,76 @@ For example, chmod -R 644 /etc/kubernetes/ssl" ``` -**Audit:** +**Audit Script:** check_files_permissions.sh ``` -/mnt/kube-bench/test_helpers/check_files_permissions.sh '/etc/kubernetes/ssl/*.pem' +#!/usr/bin/env bash + +# This script is used to ensure the file permissions are set to 644 or +# more restrictive for all files in a given directory or a wildcard +# selection of files +# +# inputs: +# $1 = /full/path/to/directory or /path/to/fileswithpattern +# ex: !(*key).pem +# +# $2 (optional) = permission (ex: 600) +# +# outputs: +# true/false + +# Turn on "extended glob" for use of '!' in wildcard +shopt -s extglob + +# Turn off history to avoid surprises when using '!' +set -H + +USER_INPUT=$1 + +if [[ "${USER_INPUT}" == "" ]]; then + echo "false" + exit +fi + + +if [[ -d ${USER_INPUT} ]]; then + PATTERN="${USER_INPUT}/*" +else + PATTERN="${USER_INPUT}" +fi + +PERMISSION="" +if [[ "$2" != "" ]]; then + PERMISSION=$2 +fi + +FILES_PERMISSIONS=$(stat -c %n\ %a ${PATTERN}) + +while read -r fileInfo; do + p=$(echo ${fileInfo} | cut -d' ' -f2) + + if [[ "${PERMISSION}" != "" ]]; then + if [[ "$p" != "${PERMISSION}" ]]; then + echo "false" + exit + fi + else + if [[ "$p" != "644" && "$p" != "640" && "$p" != "600" ]]; then + echo "false" + exit + fi + fi +done <<< "${FILES_PERMISSIONS}" + + +echo "true" +exit +``` + +**Audit Execution:** + +``` +./check_files_permissions.sh '/etc/kubernetes/ssl/*.pem' ``` **Expected result**: @@ -258,10 +348,29 @@ For example, chmod -R 600 /etc/kubernetes/ssl/certs/serverca ``` -**Audit:** +**Audit Script:** 1.1.21.sh ``` -/mnt/kube-bench/test_helpers/1.1.21.sh /etc/kubernetes/ssl +#!/bin/bash -e +check_dir=${1:-/etc/kubernetes/ssl} + +for file in $(find ${check_dir} -name "*key.pem"); do + file_permission=$(stat -c %a ${file}) + if [[ "${file_permission}" == "600" ]]; then + continue + else + echo "FAIL: ${file} ${file_permission}" + exit 666 + fi +done + +echo "pass" +``` + +**Audit Execution:** + +``` +./1.1.21.sh /etc/kubernetes/ssl ``` **Expected result**: @@ -976,10 +1085,27 @@ on the master node and set the `--encryption-provider-config` parameter to the p Follow the Kubernetes documentation and configure a `EncryptionConfig` file. In this file, choose **aescbc**, **kms** or **secretbox** as the encryption provider. -**Audit:** +**Audit Script:** 1.2.34.sh ``` -/mnt/kube-bench/test_helpers/1.2.34.sh /etc/kubernetes/ssl/encryption.yaml +#!/bin/bash -e + +check_file=${1} + +grep -q -E 'aescbc|kms|secretbox' ${check_file} +if [ $? -eq 0 ]; then + echo "--pass" + exit 0 +else + echo "fail: encryption provider found in ${check_file}" + exit 1 +fi +``` + +**Audit Execution:** + +``` +./1.2.34.sh /etc/kubernetes/ssl/encryption.yaml ``` **Expected result**: @@ -1363,10 +1489,20 @@ node and either remove the `--peer-auto-tls` parameter or set it to `false`. **Remediation:** Create an audit policy file for your cluster. -**Audit:** +**Audit Script:** 3.2.1.sh ``` -/mnt/kube-bench/test_helpers/3.2.1.sh kube-apiserver +#!/bin/bash -e + +api_server_bin=${1} + +/bin/ps -ef | /bin/grep ${api_server_bin} | /bin/grep -v ${0} | /bin/grep -v grep +``` + +**Audit Execution:** + +``` +./3.2.1.sh kube-apiserver ``` **Expected result**: @@ -1850,10 +1986,35 @@ Modify the configuration of each default service account to include this value automountServiceAccountToken: false ``` -**Audit:** +**Audit Script:** 5.1.5.sh ``` -/mnt/kube-bench/test_helpers/5.1.5.sh +#!/bin/bash + +export KUBECONFIG=${KUBECONFIG:-/root/.kube/config} + +kubectl version > /dev/null +if [ $? -gt 0 ]; then + echo "fail: kubectl failed" + exit 666 +fi + +accounts="$(kubectl --kubeconfig=${KUBECONFIG} get serviceaccounts -A -o json | jq -r '.items[] | select(.metadata.name=="default") | select((.automountServiceAccountToken == null) or (.automountServiceAccountToken == true)) | "fail \(.metadata.name) \(.metadata.namespace)"')" + +if [[ "${accounts}" == "" ]]; then + echo "--pass" + exit 0 +fi + +#echo ${accounts} +#exit 0 +echo "--pass" +``` + +**Audit Execution:** + +``` +./5.1.5.sh ``` **Expected result**: @@ -1953,10 +2114,37 @@ kubectl --kubeconfig=/root/.kube/config get psp -o json | jq .items[] | jq -r 's **Remediation:** Follow the documentation and create `NetworkPolicy` objects as you need them. -**Audit:** +**Audit Script:** 5.3.2.sh ``` -/mnt/kube-bench/test_helpers/5.3.2.sh +#!/bin/bash -e + +echo "--pass" +exit + +KUBECONFIG="/root/.kube/config" + +kubectl version > /dev/null +if [ $? -gt 0 ]; then + echo "fail: kubectl failed" + exit 666 +fi + +for namespace in $(kubectl get namespaces -A -o json | jq -r '.items[].metadata.name'); do + policy_count=$(kubectl get networkpolicy -n ${namespace} -o json | jq '.items | length') + if [ ${policy_count} -eq 0 ]; then + echo "fail: ${namespace}" + exit 666 + fi +done + +echo "--pass" +``` + +**Audit Execution:** + +``` +./5.3.2.sh ``` **Expected result**: @@ -1975,10 +2163,28 @@ Follow the documentation and create `NetworkPolicy` objects as you need them. Ensure that namespaces are created to allow for appropriate segregation of Kubernetes resources and that all new resources are created in a specific namespace. -**Audit:** +**Audit Script:** 5.6.4.sh ``` -/mnt/kube-bench/test_helpers/5.6.4.sh +#!/bin/bash -e + +export KUBECONFIG=${KUBECONFIG:-/root/.kube/config} + +kubectl version > /dev/null +if [[ $? -gt 0 ]]; then + echo "fail: kubectl failed" + exit 666 +fi + +default_resources=$(kubectl get all -o json | jq --compact-output '.items[] | select((.kind == "Service") and (.metadata.name == "kubernetes") and (.metadata.namespace == "default") | not)' | wc -l) + +echo "--count=${default_resources}" +``` + +**Audit Execution:** + +``` +./5.6.4.sh ``` **Expected result**: diff --git a/content/rancher/v2.x/en/security/hardening-2.3.5/_index.md b/content/rancher/v2.x/en/security/hardening-2.3.5/_index.md index fbf454ca480..57410298f26 100644 --- a/content/rancher/v2.x/en/security/hardening-2.3.5/_index.md +++ b/content/rancher/v2.x/en/security/hardening-2.3.5/_index.md @@ -270,7 +270,7 @@ addons: | kind: Group name: system:authenticated --- - apiVersion: extensions/v1beta1 + apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: name: restricted From 4fcf9abdee9f283e43819ad82f7cf200a13ca7b5 Mon Sep 17 00:00:00 2001 From: Nelson Roberts Date: Fri, 14 Feb 2020 16:17:10 -0700 Subject: [PATCH 12/14] EIO-247: update fixed test --- .../en/security/benchmark-2.3.5/_index.md | 20 ++++++++----------- 1 file changed, 8 insertions(+), 12 deletions(-) diff --git a/content/rancher/v2.x/en/security/benchmark-2.3.5/_index.md b/content/rancher/v2.x/en/security/benchmark-2.3.5/_index.md index 1faeb172746..864837cffe3 100644 --- a/content/rancher/v2.x/en/security/benchmark-2.3.5/_index.md +++ b/content/rancher/v2.x/en/security/benchmark-2.3.5/_index.md @@ -255,7 +255,7 @@ Run the below command (based on the file location on your system) on the master For example, ``` bash -chmod -R 644 /etc/kubernetes/ssl" +chmod -R 644 /etc/kubernetes/ssl ``` **Audit Script:** check_files_permissions.sh @@ -1994,7 +1994,7 @@ automountServiceAccountToken: false export KUBECONFIG=${KUBECONFIG:-/root/.kube/config} kubectl version > /dev/null -if [ $? -gt 0 ]; then +if [ $? -ne 0 ]; then echo "fail: kubectl failed" exit 666 fi @@ -2006,9 +2006,8 @@ if [[ "${accounts}" == "" ]]; then exit 0 fi -#echo ${accounts} -#exit 0 -echo "--pass" +echo ${accounts} +exit 666 ``` **Audit Execution:** @@ -2119,13 +2118,10 @@ Follow the documentation and create `NetworkPolicy` objects as you need them. ``` #!/bin/bash -e -echo "--pass" -exit - -KUBECONFIG="/root/.kube/config" +export KUBECONFIG=${KUBECONFIG:-"/root/.kube/config"} kubectl version > /dev/null -if [ $? -gt 0 ]; then +if [ $? -ne 0 ]; then echo "fail: kubectl failed" exit 666 fi @@ -2138,7 +2134,7 @@ for namespace in $(kubectl get namespaces -A -o json | jq -r '.items[].metadata. fi done -echo "--pass" +echo "pass" ``` **Audit Execution:** @@ -2150,7 +2146,7 @@ echo "--pass" **Expected result**: ``` -'--pass' is present +'pass' is present ``` ### 5.6 General Policies From 1b46c2749c6fa85d9366b44b4cdbe1a5fa4f1e6f Mon Sep 17 00:00:00 2001 From: Nelson Roberts Date: Fri, 14 Feb 2020 16:52:02 -0700 Subject: [PATCH 13/14] EIO-247 udpate exit code in test helper scripts --- .../v2.x/en/security/benchmark-2.3.5/_index.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/content/rancher/v2.x/en/security/benchmark-2.3.5/_index.md b/content/rancher/v2.x/en/security/benchmark-2.3.5/_index.md index 864837cffe3..aef7846d23d 100644 --- a/content/rancher/v2.x/en/security/benchmark-2.3.5/_index.md +++ b/content/rancher/v2.x/en/security/benchmark-2.3.5/_index.md @@ -360,7 +360,7 @@ for file in $(find ${check_dir} -name "*key.pem"); do continue else echo "FAIL: ${file} ${file_permission}" - exit 666 + exit 1 fi done @@ -1996,7 +1996,7 @@ export KUBECONFIG=${KUBECONFIG:-/root/.kube/config} kubectl version > /dev/null if [ $? -ne 0 ]; then echo "fail: kubectl failed" - exit 666 + exit 1 fi accounts="$(kubectl --kubeconfig=${KUBECONFIG} get serviceaccounts -A -o json | jq -r '.items[] | select(.metadata.name=="default") | select((.automountServiceAccountToken == null) or (.automountServiceAccountToken == true)) | "fail \(.metadata.name) \(.metadata.namespace)"')" @@ -2007,7 +2007,7 @@ if [[ "${accounts}" == "" ]]; then fi echo ${accounts} -exit 666 +exit 1 ``` **Audit Execution:** @@ -2123,14 +2123,14 @@ export KUBECONFIG=${KUBECONFIG:-"/root/.kube/config"} kubectl version > /dev/null if [ $? -ne 0 ]; then echo "fail: kubectl failed" - exit 666 + exit 1 fi for namespace in $(kubectl get namespaces -A -o json | jq -r '.items[].metadata.name'); do policy_count=$(kubectl get networkpolicy -n ${namespace} -o json | jq '.items | length') if [ ${policy_count} -eq 0 ]; then echo "fail: ${namespace}" - exit 666 + exit 1 fi done @@ -2169,7 +2169,7 @@ export KUBECONFIG=${KUBECONFIG:-/root/.kube/config} kubectl version > /dev/null if [[ $? -gt 0 ]]; then echo "fail: kubectl failed" - exit 666 + exit 1 fi default_resources=$(kubectl get all -o json | jq --compact-output '.items[] | select((.kind == "Service") and (.metadata.name == "kubernetes") and (.metadata.namespace == "default") | not)' | wc -l) From a02b227f55bcbd01e4ba0eb4a521823f68103a54 Mon Sep 17 00:00:00 2001 From: Nelson Roberts Date: Thu, 20 Feb 2020 11:21:20 -0700 Subject: [PATCH 14/14] EIO-247: review updates --- .../v2.x/en/security/hardening-2.3.5/_index.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/content/rancher/v2.x/en/security/hardening-2.3.5/_index.md b/content/rancher/v2.x/en/security/hardening-2.3.5/_index.md index 57410298f26..91cb760826f 100644 --- a/content/rancher/v2.x/en/security/hardening-2.3.5/_index.md +++ b/content/rancher/v2.x/en/security/hardening-2.3.5/_index.md @@ -24,7 +24,7 @@ For more detail about evaluating a hardened cluster against the official CIS ben ### Configure Kernel Runtime Parameters -The folowing `sysctl` configuration is recommended for all nodes type in the cluster. Set the following parameters in `/etc/sysctl.d/90-kubelet.conf`: +The following `sysctl` configuration is recommended for all nodes type in the cluster. Set the following parameters in `/etc/sysctl.d/90-kubelet.conf`: ``` vm.overcommit_memory=1 @@ -106,7 +106,7 @@ Once a CNI provider is enabled on a cluster a default network policy can be appl (even if policies are added that cause some pods to be treated as “isolated”), you can create a policy that explicitly allows all traffic in that namespace. Save the following `yaml` as `default-allow-all.yaml`. Additional [documentation](https://kubernetes.io/docs/concepts/services-networking/network-policies/) -about network poilices can be found on the Kubernetes site. +about network policies can be found on the Kubernetes site. > This `NetworkPolicy` is not recommended for production use @@ -136,12 +136,12 @@ done Execute this script to apply the `default-allow-all.yaml` the **permissive** `NetworkPolicy` to all namespaces. ### Reference Hardened RKE `cluster.yml` configuration -The reference `cluster.yml` is used by the RKE cli that provides the configuration needed to acheive a hardened install +The reference `cluster.yml` is used by the RKE CLI that provides the configuration needed to achieve a hardened install of Rancher Kubernetes Engine (RKE). Install [documentation](https://rancher.com/docs/rke/latest/en/installation/) is provided with additional details about the configuration items. ``` yaml -# If you intened to deploy Kubernetes in an air-gapped environment, +# If you intend to deploy Kubernetes in an air-gapped environment, # please consult the documentation on how to configure custom RKE images. kubernetes_version: "v1.15.9-rancher1-1" enable_network_policy: true @@ -417,7 +417,7 @@ dns: null ### Reference Hardened RKE Template configuration -The reference RKE Template provides the configuration needed to acheive a hardened install of Kubenetes. +The reference RKE Template provides the configuration needed to achieve a hardened install of Kubenetes. RKE Templates are used to provision Kubernetes and define Rancher settings. Follow the Rancher [documentaion](https://rancher.com/docs/rancher/v2.x/en/installation) for additional installation and RKE Template details. @@ -674,10 +674,10 @@ rancher_kubernetes_engine_config: windows_prefered_cluster: false ``` -### Hardened Reference Ubuntu **cloud-config**: +### Hardened Reference Ubuntu 18.04 LTS **cloud-config**: -The reference **cloud-config** is generenally used in cloud infrastructure environments to allow for -configuration managment of compute instances. The reference config configures Ubuntu operating system level settings +The reference **cloud-config** is generally used in cloud infrastructure environments to allow for +configuration management of compute instances. The reference config configures Ubuntu operating system level settings needed before installing kubernetes. ``` yaml