diff --git a/content/rancher/v2.x/en/faq/security/_index.md b/content/rancher/v2.x/en/faq/security/_index.md index 40ee1fae0b3..670cf73870b 100644 --- a/content/rancher/v2.x/en/faq/security/_index.md +++ b/content/rancher/v2.x/en/faq/security/_index.md @@ -6,8 +6,8 @@ weight: 8007 ### Is there a Hardening Guide? -Rancher has created a [Hardening Guide](https://releases.rancher.com/documents/security/latest/Rancher_Hardening_Guide.pdf) to provide guidance on how to set up your Rancher instance for production following security best practices. +The Hardening Guide is now located in the main [Security]({{< baseurl >}}/rancher/v2.x/en/security/) section. ### What are the results of Rancher's Kubernetes cluster when it is CIS benchmarked? -We have run the CIS Kubernetes benchmark against a hardened Rancher Kubernetes cluster and provided a [Rancher specific CIS Kubernetes benchmark self assessment](https://releases.rancher.com/documents/security/latest/Rancher_Benchmark_Assessment.pdf). +We have run the CIS Kubernetes benchmark against a hardened Rancher Kubernetes cluster. The results of that assessment can be found in the main [Security]({{< baseurl >}}/rancher/v2.x/en/security/) section. diff --git a/content/rancher/v2.x/en/security/_index.md b/content/rancher/v2.x/en/security/_index.md index 1524ce23a26..4251546a41c 100644 --- a/content/rancher/v2.x/en/security/_index.md +++ b/content/rancher/v2.x/en/security/_index.md @@ -20,7 +20,21 @@ weight: 7505 -### Rancher Vulnerabilities +### Rancher Hardening Guide + +The Rancher Hardening Guide is based off of controls and best practices found in the [CIS Kubernetes Benchmark](https://www.cisecurity.org/benchmark/kubernetes/). See Rancher's [Self Assessment of the CIS Kubernetes Benchmark](#CIS-Benchmark-Rancher-Self-Assessment) for the full list of security controls. + +* [CIS Kubernetes Benchmark 1.3.0 - Rancher 2.1.x with Kubernetes 1.11]({{< baseurl >}}/rancher/v2.x/en/security/hardening-2.1/) + +### CIS Benchmark Rancher Self-Assessment + +Since Rancher and RKE install Kubernetes services as Docker containers many of the control verification checks in the CIS Kubernetes Benchmark don't apply. This guide will walk through the various controls and provide updated example commands to audit compliance in Rancher created clusters. The original benchmark documents can be downloaded from the [CIS website](https://www.cisecurity.org/benchmark/kubernetes/). + +* [CIS Kubernetes Benchmark 1.3.0 - Rancher 2.1.x with Kubernetes 1.11]({{< baseurl >}}/rancher/v2.x/en/security/benchmark-2.1/) + +### Rancher CVEs and Resolutions + +Rancher is committed to informing the community of security issues in our products. Rancher will publish CVEs for issues we have resolved. | ID | Description | Date | Resolution | |----|-------------|------|------------| diff --git a/content/rancher/v2.x/en/security/benchmark-2.1/_index.md b/content/rancher/v2.x/en/security/benchmark-2.1/_index.md new file mode 100644 index 00000000000..bbb2f764a60 --- /dev/null +++ b/content/rancher/v2.x/en/security/benchmark-2.1/_index.md @@ -0,0 +1,1757 @@ +--- +title: Benchmark - Rancher v2.1.x +weight: 100 +--- + +### CIS Kubernetes Benchmark 1.3.0 - Rancher 2.1.x with Kubernetes 1.11 + +[Click here to download a PDF version of this document](https://releases.rancher.com/documents/security/2.1.x/Rancher_Benchmark_Assessment.pdf) + +#### Overview + +The following document scores a Kubernetes 1.11.x RKE cluster provisioned according to the Rancher 2.1.x hardening guide against the CIS 1.3.0 Kubernetes benchmark. This document is to be used by Rancher operators, security teams, auditors and decision makers. + +#### Testing controls methodology + +Rancher and RKE install Kubernetes services via Docker containers. Configuration is defined by arguments passed to the container at the time of initialization, not via configuration files. + +Scoring the commands is different in Rancher Labs than in the CIS Benchmark. Where the commands differ from the original CIS benchmark, the commands specific to Rancher Labs are provided for testing. + +When performing the tests, you will need access to the Docker command line on the hosts of all three RKE roles. The commands also make use of the the `jq` command to provide human-readable formatting. + +Tests will have an exit code of zero on success and non-zero on failure. + +#### Known Scored Control Failures + +The following scored controls do not currently pass, and Rancher Labs is working towards addressing these through future enhancements to the product. + +- 1.1.21 - Ensure that the `--kubelet-certificate-authority` argument is set as appropriate (Scored) +- 1.3.6 - Ensure that the `RotateKubeletServerCertificate` argument is set to `true` (Scored) +- 1.4.11 - Ensure that the etcd data directory permissions are set to `700` or more-restrictive (Scored) +- 1.4.12 - Ensure that the etcd data directory ownership is set to `etcd:etcd` (Scored) +- 2.1.3 - Ensure that the `--authorization-mode` argument is not set to `AlwaysAllow` (Scored) +- 2.1.9 - Ensure that the `--hostname-override` argument is not set (Scored) +- 2.1.13 - Ensure that the `--rotate-certificates` argument is not set to `false` (Scored) +- 2.1.14 - Ensure that the `RotateKubeletServerCertificate` argument is set to `true` (Scored) + +### Controls + +--- + +## 1 - Master Node Security Configuration + +### 1.1 - API Server + +#### 1.1.1 - Ensure that the `--anonymous-auth` argument is set to `false` (Scored) + +**Audit** + +``` bash +docker inspect kube-apiserver | jq -e '.[0].Args[] | match("--anonymous-auth=false").string' +``` + +**Returned Value:** `--anonymous-auth=false` + +**Result:** Pass + +#### 1.1.2 - Ensure that the `--basic-auth-file` argument is not set (Scored) + +**Audit** + +``` bash +docker inspect kube-apiserver | jq -e '.[0].Args[] | match("--basic-auth-file=.*").string' +``` + +**Returned Value:** `null` + +**Result:** Pass + +#### 1.1.3 - Ensure that the `--insecure-allow-any-token` argument is not set (Scored) + +**Audit** + +``` bash +docker inspect kube-apiserver | jq -e '.[0].Args[] | match("--insecure-allow-any-token").string' +``` + +**Returned Value:** `null` + +**Result:** Pass + +#### 1.1.4 - Ensure that the `--kubelet-https` argument is set to `true` (Scored) + +**Audit** + +``` bash +docker inspect kube-apiserver | jq -e '.[0].Args[] | match("--kubelet-https=false").string' +``` + +**Returned Value:** `null` + +**Result:** Pass + +#### 1.1.5 - Ensure that the `--insecure-bind-address` argument is not set (Scored) + +**Notes** + +Flag not set or `--insecure-bind-address=127.0.0.1`. RKE sets this flag to `--insecure-bind-address=127.0.0.1` + +**Audit** + +``` bash +docker inspect kube-apiserver | jq -e '.[0].Args[] | match("--insecure-bind-address=(?:(?!127\\.0\\.0\\.1).)+")' +``` + +**Returned Value:** `null` + +**Result:** Pass + +#### 1.1.6 - Ensure that the `--insecure-port argument` is set to `0` (Scored) + +**Audit** + +``` bash +docker inspect kube-apiserver | jq -e '.[0].Args[] | match("--insecure-port=0").string' +``` + +**Returned Value:** `--insecure-port=0` + +**Result:** Pass + +#### 1.1.7 - Ensure that the `--secure-port` argument is not set to `0` (Scored) + +**Audit** + +``` bash +docker inspect kube-apiserver | jq -e '.[0].Args[] | match("--secure-port=6443").string' +``` + +**Returned Value:** `--secure-port=6443` + +**Result:** Pass + +#### 1.1.8 - Ensure that the `--profiling` argument is set to `false` (Scored) + +**Audit** + +``` bash +docker inspect kube-apiserver | jq -e '.[0].Args[] | match("--profiling=false").string' +``` + +**Returned Value:** `--profiling=false` + +**Result:** Pass + +#### 1.1.9 - Ensure that the `--repair-malformed-updates` argument is set to `false` (Scored) + +**Audit** + +``` bash +docker inspect kube-apiserver | jq -e '.[0].Args[] | match("--repair-malformed-updates=false").string' +``` + +**Returned Value:** `--repair-malformed-updates=false` + +**Result:** Pass + +#### 1.1.10 - Ensure that the admission control plugin `AlwaysAdmit` is not set (Scored) + +**Audit** + +``` bash +docker inspect kube-apiserver | jq -e '.[0].Args[] | match("--enable-admission-plugins=.*(AlwaysAdmit).*").captures[].string' +``` + +**Returned Value:** `null` + +**Result:** Pass + +#### 1.1.11 - Ensure that the admission control plugin `AlwaysPullImages` is set (Scored) + +**Audit** + +``` bash +docker inspect kube-apiserver | jq -e '.[0].Args[] | match("--enable-admission-plugins=.*(AlwaysPullImages).*").captures[].string' +``` + +**Returned Value:** `AlwaysPullImages` + +**Result:** Pass + +#### 1.1.12 - Ensure that the admission control plugin `DenyEscalatingExec` is set (Scored) + +**Audit** + +``` bash +docker inspect kube-apiserver | jq -e '.[0].Args[] | match("--enable-admission-plugins=.*(DenyEscalatingExec).*").captures[].string' +``` + +**Returned Value:** `DenyEscalatingExec` + +**Result:** Pass + +#### 1.1.13 - Ensure that the admission control plugin `SecurityContextDeny` is set (Scored) + +**Notes** + +This control may be out of date. This **SHOULD NOT** be set if you are using `PodSecurityPolicy` (PSP). From the Kubernetes 1.11 documentation: + +> This should be enabled if a cluster doesn’t utilize pod security policies to restrict the set of values a security context can take. + +Several system services (such as `nginx-ingress`) utilize `SecurityContext` to switch users and assign capabilities. These exceptions to the general principle of not allowing privilege or capabilities can be managed with PSP. + +**Audit** + +``` bash +docker inspect kube-apiserver | jq -e '.[0].Args[] | match("--enable-admission-plugins=.*(SecurityContextDeny).*").captures[].string' +``` + +**Returned Value:** `null` + +**Result:** Document + +#### 1.1.14 - Ensure that the admission control plugin `NamespaceLifecycle` is set (Scored) + +**Audit** + +``` bash +docker inspect kube-apiserver | jq -e '.[0].Args[] | match("--enable-admission-plugins=.*(NamespaceLifecycle).*").captures[].string' +``` + +**Returned Value:** `NamespaceLifecycle` + +**Result:** Pass + +#### 1.1.15 - Ensure that the `--audit-log-path` argument is set as appropriate (Scored) + +**Notes** + +This path is the path inside of the container. It's combined with the RKE `cluster.yml` `extra-binds:` option to map the audit log to the host filesystem. + +Audit logs should be collected and shipped off-system to guarantee their integrity. + +**Audit** + +``` bash +docker inspect kube-apiserver | jq -e '.[0].Args[] | match("--audit-log-path=/var/log/kube-audit/audit-log.json").string' +``` + +**Returned Value:** `--audit-log-log=/var/log/kube-audit/audit-log.json` + +**Result:** Pass + +#### 1.1.16 - Ensure that the `--audit-log-maxage` argument is set to `30` or as appropriate (Scored) + +**Notes** + +Audit logs should be collected and shipped off-system to guarantee their integrity. Rancher Labs recommends setting this argument to a low value to prevent audit logs from filling the local disk. + +**Audit** + +``` bash +docker inspect kube-apiserver | jq -e '.[0].Args[] | match("--audit-log-maxage=\\d+").string' +``` + +**Returned Value:** `--audit-log-maxage=5` + +**Result:** Pass + +#### 1.1.17 - Ensure that the `--audit-log-maxbackup` argument is set to `10` or as appropriate (Scored) + +**Notes** + +Audit logs should be collected and shipped off-system to guarantee their integrity. Rancher Labs recommends setting this argument to a low value to prevent audit logs from filling the local disk. + +**Audit** + +``` bash +docker inspect kube-apiserver | jq -e '.[0].Args[] | match("--audit-log-maxbackup=\\d+").string' +``` + +**Returned Value:** `--audit-log-maxbackup=5` + +**Result:** Pass + +#### 1.1.18 - Ensure that the `--audit-log-maxsize` argument is set to `100` or as appropriate (Scored) + +**Notes** + +Audit logs should be collected and shipped off-system to guarantee their integrity. + +**Audit** + +``` bash +docker inspect kube-apiserver | jq -e '.[0].Args[] | match("--audit-log-maxsize=\\d+").string' +``` + +**Returned Value:** `--audit-log-maxsize=100` + +**Result:** Pass + +#### 1.1.19 - Ensure that the `--authorization-mode` argument is not set to `AlwaysAllow` (Scored) + +**Audit** + +``` +docker inspect kube-apiserver | jq -e '.[0].Args[] | match("--authorization-mode=(Node|RBAC|,)+").string' +``` + +**Returned Value:** `--authorization-mode=Node,RBAC` + +**Result:** Pass + +#### 1.1.20 - Ensure that the `--token-auth-file` parameter is not set (Scored) + +**Audit** + +``` bash +docker inspect kube-apiserver | jq -e '.[0].Args[] | match("--token-auth-file=.*").string' +``` + +**Returned Value:** `null` + +**Result:** Pass + +#### 1.1.21 - Ensure that the `--kubelet-certificate-authority` argument is set as appropriate (Scored) + +**Notes** + +RKE is using the kubelet's ability to automatically create self-signed certs. No CA cert is saved to verify the communication between `kube-apiserver` and `kubelet`. + +**Mitigation** + +Make sure nodes with `role:controlplane` are on the same local network as your nodes with `role:worker`. Use network ACLs to restrict connections to the kubelet port (10250/tcp) on worker nodes, only permitting it from controlplane nodes. + +**Audit** + +``` bash +docker inspect kube-apiserver | jq -e '.[0].Args[] | match("--kubelet-certificate-authority=.*").string' +``` + +**Returned Value:** none + +**Result:** Fail (See Mitigation) + +#### 1.1.22 - Ensure that the `--kubelet-client-certificate` and `--kubelet-client-key` arguments are set as appropriate (Scored) + +**Audit** (`--kubelet-client-certificate`) + +``` bash +docker inspect kube-apiserver | jq -e '.[0].Args[] | match("--kubelet-client-certificate=.*").string' +``` + +**Returned Value:** `--kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem` + +**Audit** (`--kubelet-client-key`) + +``` bash +docker inspect kube-apiserver | jq -e '.[0].Args[] | match("--kubelet-client-key=.*").string' +``` + +**Returned Value:** `--kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem` + +**Result:** Pass + +#### 1.1.23 Ensure that the `--service-account-lookup` argument is set to `true` (Scored) + +**Audit** + +``` bash +docker inspect kube-apiserver | jq -e '.[0].Args[] | match("--service-account-lookup=true").string' +``` + +**Returned Value:** `--service-account-lookup=true` + +**Result:** Pass + +#### 1.1.24 - Ensure that the admission control plugin `PodSecurityPolicy` is set (Scored) + +**Audit** + +``` bash +docker inspect kube-apiserver | jq -e '.[0].Args[] | match("--enable-admission-plugins=.*(PodSecurityPolicy).*").captures[].string' +``` + +**Returned Value:** `PodSecurityPolicy` + +**Result:** Pass + +#### 1.1.25 - Ensure that the `--service-account-key-file` argument is set as appropriate (Scored) + +**Audit** + +``` bash +docker inspect kube-apiserver | jq -e '.[0].Args[] | match("--service-account-key-file=.*").string' +``` + +**Returned Value:** `--service-account-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem` + +**Result:** Pass + +#### 1.1.26 - Ensure that the `--etcd-certfile` and `--etcd-keyfile` arguments are set as appropriate (Scored) + +**Audit** (`--etcd-certfile`) + +``` bash +docker inspect kube-apiserver | jq -e '.[0].Args[] | match("--etcd-certfile=.*").string' +``` + +**Returned Value:** `--etcd-certfile=/etc/kubernetes/ssl/kube-node.pem` + +**Audit** (`--etcd-keyfile`) + +``` bash +docker inspect kube-apiserver | jq -e '.[0].Args[] | match("--etcd-keyfile=.*").string' +``` + +**Returned Value:** `--etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem` + +**Result:** Pass + +#### 1.1.27 - Ensure that the admission control plugin `ServiceAccount` is set (Scored) + +**Audit** + +``` bash +docker inspect kube-apiserver | jq -e '.[0].Args[] | match("--enable-admission-plugins=.*(ServiceAccount).*").captures[].string' +``` + +**Returned Value:** `ServiceAccount` + +**Result:** Pass + +#### 1.1.28 - Ensure that the `--tls-cert-file` and `--tls-private-key-file` arguments are set as appropriate (Scored) + +**Audit** (`--tls-cert-file`) + +``` bash +docker inspect kube-apiserver | jq -e '.[0].Args[] | match("--tls-cert-file=.*").string' +``` + +**Returned Value:** `--tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem` + +**Audit** (`--tls-key-file`) + +``` bash +docker inspect kube-apiserver | jq -e '.[0].Args[] | match("--tls-private-key-file=.*").string' +``` + +**Returned Value:** `--tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem` + +**Result:** Pass + +#### 1.1.29 - Ensure that the `--client-ca-file` argument is set as appropriate (Scored) + +**Audit** + +``` bash +docker inspect kube-apiserver | jq -e '.[0].Args[] | match("--client-ca-file=.*").string' +``` + +**Returned Value:** `--client-ca-file=/etc/kubernetes/ssl/kube-ca.pem` +**Result:** Pass + +#### 1.1.30 - Ensure that the API Server only makes use of strong cryptographic ciphers (Not Scored) + +**Audit** (Allowed Ciphers) + +``` bash +docker inspect kube-apiserver | jq -e '.[0].Args[] | match("--tls-cipher-suites=.*(TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256).*").captures[].string' +``` + +**Returned Value:** `TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256` + +``` bash +docker inspect kube-apiserver | jq -e '.[0].Args[] | match("--tls-cipher-suites=.*(TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384).*").captures[].string' +``` + +**Returned Value:** `TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384` + +``` bash +docker inspect kube-apiserver | jq -e '.[0].Args[] | match("--tls-cipher-suites=.*(TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305).*").captures[].string' +``` + +**Returned Value:** `TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305` + +``` bash +docker inspect kube-apiserver | jq -e '.[0].Args[] | match("--tls-cipher-suites=.*(TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256).*").captures[].string' +``` + +**Returned Value:** + +``` bash +docker inspect kube-apiserver | jq -e '.[0].Args[] | match("--tls-cipher-suites=.*(TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384).*").captures[].string' +``` + +**Returned Value:** + +``` bash +docker inspect kube-apiserver | jq -e '.[0].Args[] | match("--tls-cipher-suites=.*(TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305).*").captures[].string' +``` + +**Returned Value:** + +``` bash +docker inspect kube-apiserver | jq -e '.[0].Args[] | match("--tls-cipher-suites=.*(TLS_RSA_WITH_AES_128_GCM_SHA256).*").captures[].string' +``` + +**Returned Value:** `TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305` + +``` bash +docker inspect kube-apiserver | jq -e '.[0].Args[] | match("--tls-cipher-suites=.*(TLS_RSA_WITH_AES_256_GCM_SHA384).*").captures[].string' +``` + +**Returned Value:** `TLS_RSA_WITH_AES_256_GCM_SHA384` + +**Audit** (Disallowed Ciphers) + +``` bash +docker inspect kube-apiserver | jq -e '.[0].Args[] | match("--tls-cipher-suites=.*(CBC).*").captures[].string' +``` + +**Returned Value:** `null` + +``` bash +docker inspect kube-apiserver | jq -e '.[0].Args[] | match("--tls-cipher-suites=.*(RC4).*").captures[].string' +``` + +**Returned Value:** `null` + +**Result:** Pass + +#### 1.1.31 - Ensure that the `--etcd-cafile` argument is set as appropriate (Scored) + +**Audit** + +``` bash +docker inspect kube-apiserver | jq -e '.[0].Args[] | match("--etcd-cafile=.*").string' +``` + +**Returned Value:** `--etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem` + +**Result:** Pass + +#### 1.1.32 - Ensure that the `--authorization-mode` argument is set to Node (Scored) + +**Audit** + +``` bash +docker inspect kube-apiserver | jq -e '.[0].Args[] | match("--authorization-mode=(Node|RBAC|,)+").string' +``` + +**Returned Value:** `--authorization-mode=Node,RBA` +**Result:** Pass + +#### 1.1.33 - Ensure that the admission control plugin `NodeRestriction` is set (Scored) + +**Audit** + +``` bash +docker inspect kube-apiserver | jq -e '.[0].Args[] | match("--enable-admission-plugins=.*(NodeRestriction).*").captures[].string' +``` + +**Returned Value:** `NodeRestriction` + +**Result:** Pass + +#### 1.1.34 - Ensure that the `--experimental-encryption-provider-config` argument is set as appropriate (Scored) + +**Audit** + +``` bash +docker inspect kube-apiserver | jq -e '.[0].Args[] | match("--experimental-encryption-provider-config=.*").string' +``` + +**Returned Value:** `--experimental-encryption-provider-config=/etc/kubernetes/encryption.yaml` + +**Result:** Pass + +#### 1.1.35 - Ensure that the encryption provider is set to aescbc (Scored) + +**Notes** + +Only the first provider in the list is active. + +**Audit** + +``` bash +grep -A 1 providers: /etc/kubernetes/encryption.yaml | grep aescbc +``` + +**Returned Value:** `- aescbc:` + +**Result:** Pass + +#### 1.1.36 - Ensure that the admission control plugin `EventRateLimit` is set (Scored) + +**Notes** + +The `EventRateLimit` plugin requires setting the `--admission-control-config-file` option and configuring details in the following files: + +- `/etc/kubernetes/admission.yaml` +- `/etc/kubernetes/event.yaml` + +See Host Configuration for details. + +**Audit** (Admissions plugin) + +``` bash +docker inspect kube-apiserver | jq -e '.[0].Args[] | match("--enable-admission-plugins=.*(EventRateLimit).*").captures[].string' +``` + +**Returned Value:** `EventRateLimit` + +**Audit** (`--admission-control-config-file`) + +``` bash +docker inspect kube-apiserver | jq -e '.[0].Args[] | match("--admission-control-config-file=.*").string' +``` + +**Returned Value:** `--admission-control-config-file=/etc/kubernetes/admission.yaml` + +**Result:** Pass + +#### 1.1.37 Ensure that the AdvancedAuditing argument is not set to false (Scored) + +**Notes** + +`AdvancedAuditing=false` should not be set, but `--audit-policy-file` should be set and configured. See Host Configuration for a sample audit policy file. + +**Audit** (Feature Gate) + +``` bash +docker inspect kube-apiserver | jq -e '.[0].Args[] | match("--feature-gates=.*(AdvancedAuditing=false).*").captures[].string' +``` + +**Returned Value:** `null` + +**Audit** (Audit Policy File) + +``` bash +docker inspect kube-apiserver | jq -e '.[0].Args[] | match("--audit-policy-file=.*").string' +``` + +**Returned Value:** `--audit-policy-file=/etc/kubernetes/audit.yaml` + +**Result:** Pass + +#### 1.1.38 Ensure that the `--request-timeout` argument is set as appropriate (Scored) + +**Notes** + +RKE uses the default value of 60s and doesn't set this option. + +**Audit** + +``` bash +docker inspect kube-apiserver | jq -e '.[0].Args[] | match("--request-timeout=.*").string' +``` + +**Returned Value:** `null` + +**Result:** Pass + +#### 1.1.39 - Ensure that the API Server only makes use of strong cryptographic ciphers (Not Scored) + +**Notes** + +This appears to be a repeat of 1.1.30. + +**Audit** (Allowed Ciphers) + +``` bash +docker inspect kube-apiserver | jq -e '.[0].Args[] | match("--tls-cipher-suites=.*(TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256).*").captures[].string' +``` + +**Returned Value:** `TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256` + +``` bash +docker inspect kube-apiserver | jq -e '.[0].Args[] | match("--tls-cipher-suites=.*(TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384).*").captures[].string' +``` + +**Returned Value:** `TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384` + +``` bash +docker inspect kube-apiserver | jq -e '.[0].Args[] | match("--tls-cipher-suites=.*(TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305).*").captures[].string' +``` + +**Returned Value:** `TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305` + +``` bash +docker inspect kube-apiserver | jq -e '.[0].Args[] | match("--tls-cipher-suites=.*(TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256).*").captures[].string' +``` + +**Returned Value:** + +``` bash +docker inspect kube-apiserver | jq -e '.[0].Args[] | match("--tls-cipher-suites=.*(TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384).*").captures[].string' +``` + +**Returned Value:** + +``` bash +docker inspect kube-apiserver | jq -e '.[0].Args[] | match("--tls-cipher-suites=.*(TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305).*").captures[].string' +``` + +**Returned Value:** + +``` bash +docker inspect kube-apiserver | jq -e '.[0].Args[] | match("--tls-cipher-suites=.*(TLS_RSA_WITH_AES_128_GCM_SHA256).*").captures[].string' +``` + +**Returned Value:** `TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305` + +``` bash +docker inspect kube-apiserver | jq -e '.[0].Args[] | match("--tls-cipher-suites=.*(TLS_RSA_WITH_AES_256_GCM_SHA384).*").captures[].string' +``` + +**Returned Value:** `TLS_RSA_WITH_AES_256_GCM_SHA384` + +**Audit** (Disallowed Ciphers) + +``` bash +docker inspect kube-apiserver | jq -e '.[0].Args[] | match("--tls-cipher-suites=.*(CBC).*").captures[].string' +``` + +**Returned Value:** `null` + +``` bash +docker inspect kube-apiserver | jq -e '.[0].Args[] | match("--tls-cipher-suites=.*(RC4).*").captures[].string' +``` + +**Returned Value:** `null` + +**Result:** Pass + +### 1.2 - Scheduler + +#### 1.2.1 - Ensure that the `--profiling` argument is set to false (Scored) + +**Audit** + +``` bash +docker inspect kube-scheduler | jq -e '.[0].Args[] | match("--profiling=false").string' +``` + +**Returned Value:** `--profiling=false` +**Result:** Pass + +#### 1.2.2 - Ensure that the `--address` argument is set to 127.0.0.1 (Scored) + +**Audit** + +``` bash +docker inspect kube-scheduler | jq -e '.[0].Args[] | match("--address=127\\.0\\.0\\.1").string' +``` + +**Returned Value:** `--address=127.0.0.1` +**Result:** Pass + +### 1.3 - Controller Manager + +#### 1.3.1 - Ensure that the `--terminated-pod-gc-threshold` argument is set as appropriate (Scored) + +**Audit** + +``` bash +docker inspect kube-controller-manager | jq -e '.[0].Args[] | match("--terminated-pod-gc-threshold=\\d+").string' +``` + +**Returned Value:** `--terminated-pod-gc-threshold=1000` +**Result:** Pass + +#### 1.3.2 - Ensure that the `--profiling` argument is set to false (Scored) + +**Audit** + +``` bash +docker inspect kube-controller-manager | jq -e '.[0].Args[] | match("--profiling=false").string' +``` + +**Returned Value:** `--profiling=false` + +**Result:** Pass + +#### 1.3.3 - Ensure that the `--use-service-account-credentials` argument is set to true (Scored) + +**Audit** + +``` bash +docker inspect kube-controller-manager | jq -e '.[0].Args[] | match("--use-service-account-credentials=true").string' +``` + +**Returned Value:** `--use-service-account-credentials=true` + +**Result:** Pass + +#### 1.3.4 - Ensure that the `--service-account-private-key-file` argument is set as appropriate (Scored) + +**Audit** + +``` bash +docker inspect kube-controller-manager | jq -e '.[0].Args[] | match("--service-account-private-key-file=.*").string' +``` + +**Returned Value:** `--service-account-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem` + +**Result:** Pass + +#### 1.3.5 - Ensure that the `--root-ca-file` argument is set as appropriate (Scored) + +**Audit** + +``` bash +docker inspect kube-controller-manager | jq -e '.[0].Args[] | match("--root-ca-file=.*").string' +``` + +**Returned Value:** `--root-ca-file=/etc/kubernetes/ssl/kube-ca.pem` + +**Result:** Pass + +#### 1.3.6 - Ensure that the RotateKubeletServerCertificate argument is set to true (Scored) + +**Notes** + +RKE does not yet support certificate rotation. This feature is due for the 0.1.12 release of RKE. + +**Audit** + +``` bash +docker inspect kube-apiserver | jq -e '.[0].Args[] | match("--feature-gates=.*(RotateKubeletServerCertificate=true).*").captures[].string' +``` + +**Returned Value:** `null` + +**Result:** Fail + +#### 1.3.7 - Ensure that the `--address` argument is set to 127.0.0.1 (Scored) + +**Audit** + +``` bash +docker inspect kube-controller-manager | jq -e '.[0].Args[] | match("--address=127\\.0\\.0\\.1").string' +``` + +**Returned Value:** `--address=127.0.0.1` + +**Result:** Pass + +### 1.4 - Configuration Files + +#### 1.4.1 - Ensure that the API server pod specification file permissions are set to 644 or more restrictive (Scored) + +**Notes** + +RKE doesn't require or maintain a configuration file for kube-apiserver. All configuration is passed in as arguments at container run time. + +**Result:** Pass (Not Applicable) + +#### 1.4.2 - Ensure that the API server pod specification file ownership is set to `root:root` (Scored) + +**Notes** + +RKE doesn't require or maintain a configuration file for kube-apiserver. All configuration is passed in as arguments at container run time. + +**Result:** Pass (Not Applicable) + +#### 1.4.3 - Ensure that the controller manager pod specification file permissions are set to `644` or more restrictive (Scored) + +**Notes** + +RKE doesn't require or maintain a configuration file for `kube-controller-manager`. All configuration is passed in as arguments at container run time. + +**Result:** Pass (Not Applicable) + +#### 1.4.4 - Ensure that the controller manager pod specification file ownership is set to `root:root` (Scored) + +**Notes** + +RKE doesn't require or maintain a configuration file for `kube-controller-manager`. All configuration is passed in as arguments at container run time. + +**Result:** Pass (Not Applicable) + +#### 1.4.5 - Ensure that the scheduler pod specification file permissions are set to `644` or more restrictive (Scored) + +**Notes** + +RKE doesn't require or maintain a configuration file for `kube-scheduler`. All configuration is passed in as arguments at container run time. + +**Result:** Pass (Not Applicable) + +#### 1.4.6 - Ensure that the scheduler pod specification file ownership is set to `root:root` (Scored) + +**Notes** + +RKE doesn't require or maintain a configuration file for kube-scheduler. All configuration is passed in as arguments at container run time. + +**Result:** Pass (Not Applicable) + +#### 1.4.7 - Ensure that the `etcd` pod specification file permissions are set to `644` or more restrictive (Scored) + +**Notes** + +RKE doesn't require or maintain a configuration file for etcd. All configuration is passed in as arguments at container run time. + +**Result:** Pass (Not Applicable) + +#### 1.4.8 - Ensure that the `etcd` pod specification file ownership is set to `root:root` (Scored) + +**Notes** + +RKE doesn't require or maintain a configuration file for etcd. All configuration is passed in as arguments at container run time. + +**Result:** Pass (Not Applicable) + +#### 1.4.9 - Ensure that the Container Network Interface file permissions are set to `644` or more restrictive (Not Scored) + +**Notes** + +This is a manual check. + +**Audit** (`/var/lib/cni/networks/k8s-pod-network`) + +``` bash +ls -l /var/lib/cni/networks/k8s-pod-network/ +``` + +**Returned Value:** + +``` bash +-rw-r--r-- 1 root root 64 Nov 6 20:05 10.42.0.2 +-rw-r--r-- 1 root root 64 Nov 6 20:05 10.42.0.3 +-rw-r--r-- 1 root root 64 Nov 6 20:05 10.42.0.4 +-rw-r--r-- 1 root root 64 Nov 6 20:05 10.42.0.5 +-rw-r--r-- 1 root root 10 Nov 7 16:25 last_reserved_ip.0 +``` + +**Audit** (`/etc/cni/net.d`) + +``` bash +ls -l /etc/cni/net.d/ +``` + +**Returned Value:** + +``` bash +-rw-r--r-- 1 root root 1474 Nov 6 20:05 10-calico.conflist +-rw------- 1 root root 2529 Nov 6 20:05 calico-kubeconfig +``` + +**Result:** Pass + +#### 1.4.10 - Ensure that the Container Network Interface file ownership is set to `root:root` (Not Scored) + +**Notes** + +This is a manual check. + +**Audit** (`/var/lib/cni/networks/k8s-pod-network`) + +``` bash +ls -l /var/lib/cni/networks/k8s-pod-network/ +``` + +**Returned Value:** + +``` bash +-rw-r--r-- 1 root root 64 Nov 6 20:05 10.42.0.2 +-rw-r--r-- 1 root root 64 Nov 6 20:05 10.42.0.3 +-rw-r--r-- 1 root root 64 Nov 6 20:05 10.42.0.4 +-rw-r--r-- 1 root root 64 Nov 6 20:05 10.42.0.5 +-rw-r--r-- 1 root root 10 Nov 7 16:25 last_reserved_ip.0 +``` + +**Audit** (`/etc/cni/net.d`) + +``` bash +ls -l /etc/cni/net.d/ +``` + +**Returned Value:** + +``` bash +-rw-r--r-- 1 root root 1474 Nov 6 20:05 10-calico.conflist +-rw------- 1 root root 2529 Nov 6 20:05 calico-kubeconfig +``` + +**Result:** Pass + +#### 1.4.11 - Ensure that the etcd data directory permissions are set to 700 or more restrictive (Scored) + +**Notes** + +Files underneath the data dir are permissioned `700` + +``` bash +ls -al /var/lib/etcd +total 12 +drwxr-xr-x 3 root root 4096 Nov 14 17:06 . +drwxr-xr-x 47 root root 4096 Nov 14 17:06 .. +drwx------ 4 root root 4096 Nov 14 17:06 member +``` + +**Audit** + +``` bash +stat -c %a /var/lib/etcd +``` + +**Returned Value:** `755` + +**Result:** Fail + +#### 1.4.12 - Ensure that the `etcd` data directory ownership is set to `etcd:etcd` (Scored) + +**Notes** + +The `etcd` container runs as the `root` user. The data directory and files are owned by `root`. + +**Audit** + +``` bash +stat -c %U:%G /var/lib/etcd +``` + +**Returned Value:** `root:root` + +**Result:** Fail + +#### 1.4.13 - Ensure that the file permissions for `admin.conf` are set to `644` or more restrictive (Scored) + +**Notes** + +RKE does not store the kubernetes default kubeconfig credentials file on the nodes. It's presented to user where RKE is run. We recommend that this kube_config_cluster.yml file be kept in secure store. + +**Result:** Pass (Not Applicable) + +#### 1.4.14 - Ensure that ownership of `admin.conf` is set to `root:root` (Scored) + +**Notes** + +RKE does not store the default `kubectl` config credentials file on the nodes. It presents credentials to the user when `rke` is first run, and only on the device where the user ran the command. Rancher Labs recommends that this `kube_config_cluster.yml` file be kept in secure store. + +**Result:** Pass (Not Applicable) + +#### 1.4.15 - Ensure that the file permissions for `scheduler.conf` are set to `644` or more restrictive (Scored) + +**Audit** + +``` bash +stat -c %a /etc/kubernetes/ssl/kubecfg-kube-scheduler.yaml +``` + +**Returned Value:** `644` + +**Result:** Pass + +#### 1.4.16 - Ensure that the file ownership of `scheduler.conf` is set to `root:root` (Scored) + +**Audit** + +``` bash +stat -c %U:%G /etc/kubernetes/ssl/kubecfg-kube-scheduler.yaml +``` + +**Returned Value:** `root:root` + +**Result:** Pass + +#### 1.4.17 - Ensure that the file permissions for `controller-manager.conf` are set to `644` or more restrictive (Scored) + +**Audit** + +``` bash +stat -c %a /etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml +``` + +**Returned Value:** `644` + +**Result:** Pass + +#### 1.4.18 - Ensure that the file ownership of `controller-manager.conf` is set to `root:root` (Scored) + +**Audit** + +``` bash +stat -c %U:%G /etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml +``` + +**Returned Value:** `root:root` + +**Result:** Pass + +### 1.5 - etcd + +#### 1.5.1 - Ensure that the `--cert-file` and `--key-file` arguments are set as appropriate (Scored) + +**Audit** `(--cert-file`) + +``` bash +docker inspect etcd | jq -e '.[0].Args[] | match("--cert-file=.*").string' +``` + +**Returned Value:** `--cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-22-135.pem` + +**Audit** (`--key-file`) + +``` bash +docker inspect etcd | jq -e '.[0].Args[] | match("--key-file=.*").string' +``` + +**Returned Value:** `--key-file=/etc/kubernetes/ssl/kube-etcd-172-31-22-135-key.pem` + +**Result:** Pass + +#### 1.5.2 - Ensure that the `--client-cert-auth` argument is set to `true` (Scored) + +**Notes** + +Setting "--client-cert-auth" is the equivalent of setting "--client-cert-auth=true". + +**Audit** + +``` bash +docker inspect etcd | jq -e '.[0].Args[] | match("--client-cert-auth(=true)*").string' +``` + +**Returned Value:** `--client-cert-auth` + +**Result:** Pass + +#### 1.5.3 - Ensure that the `--auto-tls` argument is not set to `true` (Scored) + +**Audit** + +``` bash +docker inspect etcd | jq -e '.[0].Args[] | match("--auto-tls(?:(?!=false).*)").string' +``` + +**Returned Value:** `null` + +**Result:** Pass + +#### 1.5.4 - Ensure that the `--peer-cert-file` and `--peer-key-file` arguments are set as appropriate (Scored) + +**Audit** (`--peer-cert-file) + +``` bash +docker inspect etcd | jq -e '.[0].Args[] | match("--peer-cert-file=.*").string' +``` + +**Returned Value:** `--peer-cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-22-135.pem` + +**Audit** (`--peer-key-file) + +``` bash +docker inspect etcd | jq -e '.[0].Args[] | match("--peer-key-file=.*").string' +``` + +**Returned Value:** `--peer-key-file=/etc/kubernetes/ssl/kube-etcd-172-31-22-135-key.pem` + +**Result:** Pass + +#### 1.5.5 - Ensure that the `--peer-client-cert-auth` argument is set to `true` (Scored) + +**Notes** + +Setting "--peer-client-cert-auth" is the equivalent of setting "--peer-client-cert-auth=true". + +**Audit** + +``` bash +docker inspect etcd | jq -e '.[0].Args[] | match("--peer-client-cert-auth(=true)*").string' +``` + +**Returned Value:** `--client-cert-auth` + +**Result:** Pass + +#### 1.5.6 - Ensure that the `--peer-auto-tls` argument is not set to `true` (Scored) + +**Audit** + +``` bash +docker inspect etcd | jq -e '.[0].Args[] | match("--peer-auto-tls(?:(?!=false).*)").string' +``` + +**Returned Value:** `null` + +**Result:** Pass + +#### 1.5.7 - Ensure that a unique Certificate Authority is used for `etcd` (Not Scored) + +**Notes** + +RKE does not currently implement a seperate CA for etcd certificates. + +`--trusted-ca-file` is set and different from the `--client-ca-file` used by `kube-apiserver`. + +**Audit** + +``` bash +docker inspect etcd | jq -e '.[0].Args[] | match("--trusted-ca-file=(?:(?!/etc/kubernetes/ssl/kube-ca.pem).*)").string' +``` + +**Returned Value:** `null` + +**Result:** Fail + +#### 1.6 - General Security Primitives + +These "Not Scored" controls are implementation best practices. To ease the administrative burden, we recommend that you implement these best practices on your workload clusters by creating clusters with Rancher rather than using RKE alone. + +#### 1.6.1 - Ensure that the cluster-admin role is only used where required (Not Scored) + + +Rancher has built in support for maintaining and enforcing Kubernetes RBAC on your workload clusters. + +Rancher has the ability integrate with external authentication sources (LDAP, SAML, AD…) allows easy access with unique credentials to your existing users or groups. + +#### 1.6.2 - Create administrative boundaries between resources using namespaces (Not Scored) + +With Rancher, users or groups can be assigned access to all clusters, a single cluster or a "Project" (a group of one or more namespaces in a cluster). This allows granular access control to cluster resources. + +#### 1.6.3 - Create network segmentation using Network Policies (Not Scored) + +Rancher can (optionally) automatically create Network Policies to isolate "Projects" (a group of one or more namespaces) in a cluster. + +See "Cluster Options" when creating a cluster with Rancher to turn on Network Isolation. + +#### 1.6.4 - Ensure that the `seccomp` profile is set to `docker/default` in your pod definitions (Not Scored) + +Since this requires the enabling of AllAlpha feature gates we would not recommend enabling this feature at the moment. + +#### 1.6.5 - Apply security context to your pods and containers (Not Scored) + +This practice does go against control 1.1.13, but we prefer using PSP and allowing security context to be set over a blanket deny. + +Rancher allows users to set various Security Context options when launching pods via the GUI interface. + +#### 1.6.6 - Configure image provenance using the `ImagePolicyWebhook` admission controller (Not Scored) + +Image Policy Webhook requires a 3rd party service to enforce policy. This can be configured in the `--admission-control-config-file`. See the Host configuration section for the admission.yaml file. + +#### 1.6.7 - Configure network policies as appropriate (Not Scored) + +Rancher can (optionally) automatically create Network Policies to isolate projects (a group of one or more namespaces) within a cluster. + +See the _Cluster Options_ section when creating a cluster with Rancher to turn on network isolation. + +#### 1.6.8 - Place compensating controls in the form of PSP and RBAC for privileged container usage (Not Scored) + +Section 1.7 of this guide shows how to add and configure a default "restricted" PSP based on controls. + +With Rancher you can create a centrally maintained "restricted" PSP and deploy it to all of the clusters that Rancher manages. + + +#### 1.7 - Pod Security Policies (PSP) + +This RKE configuration has two Pod Security Policies. + +- `default-psp`: assigned to namespaces that require additional privileged access: `kube-system`, `ingress-nginx` and `cattle-system`. +- `restricted`: This is the cluster default PSP and follows the best practices defined by controls in this section. + +#### 1.7.1 - Do not admit privileged containers (Not Scored) + +**Notes** + +The restricted PSP is available to all ServiceAccounts. + +**Audit** + +``` bash +kubectl get psp restricted -o jsonpath='{.spec.privileged}' | grep "true" +``` + +**Returned Value:** `null` + +**Result:** Pass + +#### 1.7.2 - Do not admit containers wishing to share the host process ID namespace (Scored) + +**Notes** + +The restricted PSP is available to all ServiceAccounts. + +**Audit** + +``` bash +kubectl get psp restricted -o jsonpath='{.spec.hostPID}' | grep "true" +``` + +**Returned Value:** `null` + +**Result:** Pass + +#### 1.7.3 - Do not admit containers wishing to share the host IPC namespace (Scored) + +**Notes** + +The restricted PSP is available to all ServiceAccounts. + +**Audit** + +``` bash +kubectl get psp restricted -o jsonpath='{.spec.hostIPC}' | grep "true" +``` + +**Returned Value:** `null` + +**Result:** Pass + +#### 1.7.4 - Do not admit containers wishing to share the host network namespace (Scored) + +**Notes** + +The restricted PSP is available to all ServiceAccounts. + +**Audit** + +``` bash +kubectl get psp restricted -o jsonpath='{.spec.hostNetwork}' | grep "true" +``` + +**Returned Value:** `null` + +**Result:** Pass + +#### 1.7.5 - Do not admit containers with `allowPrivilegeEscalation` (Scored) + +**Notes** + +The restricted PSP is available to all ServiceAccounts. + +**Audit** + +``` bash +kubectl get psp restricted -o jsonpath='{.spec.allowPrivilegeEscalation}' | grep "true" +``` + +**Returned Value:** `null` + +**Result:** Pass + +#### 1.7.6 - Do not admit containers whose processes run as `root` (Not Scored) + +**Notes** + +The restricted PSP is available to all ServiceAccounts. + +**Audit** + +``` bash +kubectl get psp restricted -o jsonpath='{.spec.runAsUser.rule}' | grep "RunAsAny" +``` + +**Returned Value:** `null` + +**Result:** Pass + +#### 1.7.7 - Do not admit containers with dangerous capabilities (Not Scored) + +**Notes** + +The restricted PSP is available to all ServiceAccounts. + +**Audit** + +``` bash +kubectl get psp restricted -o jsonpath='{.spec.requiredDropCapabilities}' | grep "NET_RAW" +``` + +**Returned Value:** `[NET_RAW]` + +**Result:** Pass + +## 2 - Worker Node Security Configuration + +### 2.1 - Kubelet + +#### 2.1.1 - Ensure that the `--allow-privileged` argument is set to false (Scored) + +**Notes** + +The `--allow-privileged` argument is deprecated from Kubernetes v1.11, and the default setting is `true` with the intention that users should use `PodSecurityPolicy` settings to allow or prevent privileged containers. + +Our RKE configuration uses `PodSecurityPolicy` with a default policy to reject privileged containers. + +**Result:** Pass (Not Applicable) + +#### 2.1.2 - Ensure that the `--anonymous-auth` argument is set to `false` (Scored) + +**Audit** + +``` bash +docker inspect kubelet | jq -e '.[0].Args[] | match("--anonymous-auth=false").string' +``` + +**Returned Value:** `--anonymous-auth=false` + +**Result:** Pass + +#### 2.1.3 - Ensure that the `--authorization-mode` argument is not set to `AlwaysAllow` (Scored) + +**Notes** + +RKE currently runs the kubelet without the `--authorization-mode` flag. + +**Audit** + +``` bash +docker inspect kubelet | jq -e '.[0].Args[] | match("--authorization-mode=Webhook").string' +``` + +**Returned Value:** `null` + +**Result:** Fail + +#### 2.1.4 - Ensure that the `--client-ca-file` argument is set as appropriate (Scored) + +**Audit** + +``` bash +docker inspect kubelet | jq -e '.[0].Args[] | match("--client-ca-file=.*").string' +``` + +**Returned Value:** `--client-ca-file=/etc/kubernetes/ssl/kube-ca.pem` + +**Result:** Pass + +#### 2.1.5 - Ensure that the `--read-only-port` argument is set to `0` (Scored) + +**Audit** + +``` bash +docker inspect kubelet | jq -e '.[0].Args[] | match("--read-only-port=0").string' +``` + +**Returned Value:** `--read-only-port=0` + +**Result:** Pass + +#### 2.1.6 - Ensure that the `--streaming-connection-idle-timeout` argument is not set to `0` (Scored) + +**Audit** + +``` bash +docker inspect kubelet | jq -e '.[0].Args[] | match("--streaming-connection-idle-timeout=.*").string' +``` + +**Returned Value:** `--streaming-connection-idle-timeout=1800s` + +**Result:** Pass + +#### 2.1.7 - Ensure that the `--protect-kernel-defaults` argument is set to `true` (Scored) + +**Audit** + +``` bash +docker inspect kubelet | jq -e '.[0].Args[] | match("--protect-kernel-defaults=true").string' +``` + +**Returned Value:** `--protect-kernel-defaults=true` + +**Result:** Pass + +#### 2.1.8 - Ensure that the `--make-iptables-util-chains` argument is set to `true` (Scored) + +**Audit** + +``` bash +docker inspect kubelet | jq -e '.[0].Args[] | match("--make-iptables-util-chains=true").string' +``` + +**Returned Value:** `--make-iptables-util-chains=true` + +**Result:** Pass + +#### 2.1.9 - Ensure that the `--hostname-override` argument is not set (Scored) + +**Audit** + +``` bash +docker inspect kubelet | jq -e '.[0].Args[] | match("--hostname-override=.*").string' +``` + +**Returned Value:** `--hostname-override=` + +**Result:** Fail + +#### 2.1.10 - Ensure that the `--event-qps` argument is set to `0` (Scored) + +**Audit** + +``` bash +docker inspect kubelet | jq -e '.[0].Args[] | match("--event-qps=0").string' +``` + +**Returned Value:** `--event-qps=0` + +**Result:** Pass + +#### 2.1.11 - Ensure that the `--tls-cert-file` and `--tls-private-key-file` arguments are set as appropriate (Scored) + +**Notes** + +RKE does not set these options and uses the kubelet's self generated certificates for TLS communication. These files are located in the default directory (`/var/lib/kubelet/pki`). + +**Audit** (`--tls-cert-file`) + +``` bash +docker inspect kubelet | jq -e '.[0].Args[] | match("--tls-cert-file=.*").string' +``` + +**Returned Value:** `null` + +**Audit** (`--tls-private-key-file`) + +``` bash +docker inspect kubelet | jq -e '.[0].Args[] | match("--tls-private-key-file=.*").string' +``` + +**Returned Value:** `null` + +**Result:** Pass + +#### 2.1.12 - Ensure that the `--cadvisor-port` argument is set to `0` (Scored) + +**Audit** + +``` bash +docker inspect kubelet | jq -e '.[0].Args[] | match("--cadvisor-port=0").string' +``` + +**Returned Value:** `--cadvisor-port=0` + +**Result:** Pass + +#### 2.1.13 - Ensure that the `--rotate-certificates` argument is not set to `false` (Scored) + +**Notes** + +RKE will enable certificate rotation in version 0.1.12. + +**Audit** + +``` bash +docker inspect kubelet | jq -e '.[0].Args[] | match("--rotate-certificates=true").string' +``` + +**Returned Value:** `null` + +**Result:** Fail + +#### 2.1.14 - Ensure that the `RotateKubeletServerCertificate` argument is set to `true` (Scored) + +**Notes** + +RKE does not yet support certificate rotation. This feature is due for the 0.1.12 release of RKE. + +**Audit** + +``` bash +docker inspect kubelet | jq -e '.[0].Args[] | match("--feature-gates=.*(RotateKubeletServerCertificate=true).*").captures[].string' +``` + +**Returned Value:** `null` + +**Result:** Fail + +#### 2.1.15 - Ensure that the kubelet only makes use of strong cryptographic ciphers (Not Scored) + +**Audit** (Allowed Ciphers) + +``` bash +docker inspect kubelet | jq -e '.[0].Args[] | match("--tls-cipher-suites=.*(TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256).*").captures[].string' +``` + +**Returned Value:** `TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256` + +**Audit** + +``` bash +docker inspect kubelet | jq -e '.[0].Args[] | match("--tls-cipher-suites=.*(TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384).*").captures[].string' +``` + +**Returned Value:** `TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384` + +**Audit** + +``` bash +docker inspect kubelet | jq -e '.[0].Args[] | match("--tls-cipher-suites=.*(TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305).*").captures[].string' +``` + +**Returned Value:** `TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305` + +**Audit** + +``` bash +docker inspect kubelet | jq -e '.[0].Args[] | match("--tls-cipher-suites=.*(TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256).*").captures[].string' +``` + +**Returned Value:** + +**Audit** + +``` bash +docker inspect kubelet | jq -e '.[0].Args[] | match("--tls-cipher-suites=.*(TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384).*").captures[].string' +``` + +**Returned Value:** + +**Audit** + +``` bash +docker inspect kubelet | jq -e '.[0].Args[] | match("--tls-cipher-suites=.*(TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305).*").captures[].string' +``` + +**Returned Value:** + +**Audit** + +``` bash +docker inspect kubelet | jq -e '.[0].Args[] | match("--tls-cipher-suites=.*(TLS_RSA_WITH_AES_128_GCM_SHA256).*").captures[].string' +``` + +**Returned Value:** `TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305` + +**Audit** + +``` bash +docker inspect kubelet | jq -e '.[0].Args[] | match("--tls-cipher-suites=.*(TLS_RSA_WITH_AES_256_GCM_SHA384).*").captures[].string' +``` + +**Returned Value:** `TLS_RSA_WITH_AES_256_GCM_SHA384` + +**Audit** (Disallowed Ciphers) + +``` bash +docker inspect kubelet | jq -e '.[0].Args[] | match("--tls-cipher-suites=.*(CBC).*").captures[].string' +``` + +**Returned Value:** `null` + +**Audit** + +``` bash +docker inspect kubelet | jq -e '.[0].Args[] | match("--tls-cipher-suites=.*(RC4).*").captures[].string' +``` + +**Returned Value:** `null` + +**Result:** Pass + +### 2.2 - Configuration Files + +#### 2.2.1 - Ensure that the permissions for `kubelet.conf` are set to `644` or more restrictive (Scored) + +**Notes** + +This is the value of the `--kubeconfig` option. + +**Audit** + +``` bash +stat -c %a /etc/kubernetes/ssl/kubecfg-kube-node.yaml +``` + +**Returned Value:** `644` + +**Result:** Pass + +#### 2.2.2 - Ensure that the kubelet.conf file ownership is set to root:root (Scored) + +**Notes** + +This is the value of the `--kubeconfig` option. + +**Audit** + +``` bash +stat -c %U:%G /etc/kubernetes/ssl/kubecfg-kube-node.yaml +``` + +**Returned Value:** `root:root` + +**Result:** Pass + +#### 2.2.3 - Ensure that the kubelet service file permissions are set to `644` or more restrictive (Scored) + +**Notes** + +RKE doesn't require or maintain a configuration file for kubelet. All configuration is passed in as arguments at container run time. + +**Result:** Pass (Not Applicable) + + +#### 2.2.4 - Ensure that the kubelet service file ownership is set to `root:root` (Scored) + +**Notes** + +RKE doesn't require or maintain a configuration file for kubelet. All configuration is passed in as arguments at container run time. + +**Result:** Pass (Not Applicable) + +#### 2.2.5 - Ensure that the proxy kubeconfig file permissions are set to `644` or more restrictive (Scored) + +**Audit** + +``` bash +stat -c %a /etc/kubernetes/ssl/kubecfg-kube-proxy.yaml +``` + +**Returned Value:** `644` + +**Result:** Pass + +#### 2.2.6 - Ensure that the proxy kubeconfig file ownership is set to `root:root` (Scored) + +**Audit** + +``` bash +stat -c %U:%G /etc/kubernetes/ssl/kubecfg-kube-proxy.yaml +``` + +**Returned Value:** `root:root` + +**Result:** Pass + +#### 2.2.7 - Ensure that the certificate authorities file permissions are set to `644` or more restrictive (Scored) + +**Audit** + +``` bash +stat -c %a /etc/kubernetes/ssl/kube-ca.pem +``` + +**Returned Value:** `644` + +**Result:** Pass + +#### 2.2.8 - Ensure that the client certificate authorities file ownership is set to `root:root` (Scored) + +**Audit** + +``` bash +stat -c %U:%G /etc/kubernetes/ssl/kube-ca.pem +``` + +**Returned Value:** `root:root` + +**Result:** Pass + +#### 2.2.9 - Ensure that the kubelet configuration file ownership is set to `root:root` (Scored) + +**Notes** + +RKE doesn't require or maintain a configuration file for kubelet. All configuration is passed in as arguments at container run time. + +**Result:** Pass (Not Applicable) + +#### 2.2.10 - Ensure that the kubelet configuration file permissions are set to `644` or more restrictive (Scored) + +**Notes** + +RKE doesn't require or maintain a configuration file for kubelet. All configuration is passed in as arguments at container run time. + +**Result:** Pass (Not Applicable) diff --git a/content/rancher/v2.x/en/security/hardening-2.1/_index.md b/content/rancher/v2.x/en/security/hardening-2.1/_index.md new file mode 100644 index 00000000000..1883b860709 --- /dev/null +++ b/content/rancher/v2.x/en/security/hardening-2.1/_index.md @@ -0,0 +1,1170 @@ +--- +title: Hardening Guide - Rancher v2.1.x +weight: 100 +--- + +### Hardening Guide for Rancher 2.1.x with Kubernetes 1.11 + +[Click here to download a PDF version of this document](https://releases.rancher.com/documents/security/2.1.x/Rancher_Hardening_Guide.pdf) + +### Overview + +This document provides prescriptive guidance for hardening a production installation of Rancher v2.1.x. It outlines the configurations and controls required to address CIS-Kubernetes benchmark controls. + +[CIS Benchmark Rancher Self-Assessment Guide]({{< baseurl >}}/rancher/v2.x/en/security/benchmark-2.1/) + +### Profile Definitions + +The following profile definitions agree with the CIS Benchmarks for Kubernetes. + +#### Level 1 + +Items in this profile intend to: + +- offer practical advice appropriate for the environment; +- deliver an obvious security benefit; and +- not alter the functionality or utility of the environment beyond an acceptable margin + +#### Level 2 + +Items in this profile extend the “Level 1” profile and exhibit one or more of the following characteristics: + +- are intended for use in environments or use cases where security is paramount +- act as a defense in depth measure +- may negatively impact the utility or performance of the technology + +--- + +## 1.1 - Rancher HA Kubernetes cluster host configuration + +### 1.1.1 - Configure default sysctl settings on all hosts + +**Profile Applicability** + +- Level 1 + +**Description** + +Configure sysctl settings to match what the kubelet would set if allowed. + +**Rationale** + +We recommend that users launch the kubelet with the `--protect-kernel-defaults` option. The settings that the kubelet initially attempts to change can be set manually. + +This supports the following control: + +- 2.1.7 - Ensure that the `--protect-kernel-defaults` argument is set to true (Scored) + +**Audit** + +- Verify `vm.overcommit_memory = 1` + +``` bash +sysctl vm.overcommit_memory +``` + +- Verify `kernel.panic = 10` + +``` bash +sysctl kernel.panic +``` + +- Verify `kernel.panic_on_oops = 1` + +``` bash +sysctl kernel.panic_on_oops +``` + +**Remediation** + +- Set the following parameters in `/etc/sysctl.conf` on all nodes: + +``` plain +vm.overcommit_memory=1 +kernel.panic=10 +kernel.panic_on_oops=1 +``` + +- Run `sysctl -p` to enable the settings. + +### 1.1.2 - Install the encryption provider configuration on all control plane nodes + +**Profile Applicability** + +- Level 1 + +**Description** + +Create a Kubernetes encryption configuration file on each of the RKE nodes that will be provisioned with the `controlplane` role: + +**Rationale** + +This configuration file will ensure that the Rancher RKE cluster encrypts secrets at rest, which Kubernetes does not do by default. + +This supports the following controls: + +- 1.1.34 - Ensure that the `--experimental-encryption-provider-config` argument is set as appropriate (Scored) +- 1.1.35 - Ensure that the encryption provider is set to `aescbc` (Scored) + +**Audit** + +On the control plane hosts for the Rancher HA cluster run: + +``` bash +stat /etc/kubernetes/encryption.yaml +``` + +Ensure that: + +- The file is present +- The file mode is `0600` +- The file owner is `root:root` +- The file contains: + +``` yaml +apiVersion: v1 +kind: EncryptionConfig +resources: +- resources: + - secrets + providers: + - aescbc: + keys: + - name: key1 + secret: <32-byte base64 encoded string> + - identity: {} +``` + +Where `aescbc` is the key type, and `secret` is populated with a 32-byte base64 encoded string. + +**Remediation** + +- Generate a key and an empty configuration file: + +``` bash +head -c 32 /dev/urandom | base64 -i - +touch /etc/kubernetes/encryption.yaml +``` + +- Set the file ownership to `root:root` and the permissions to `0600` + +``` bash +chown root:root /etc/kubernetes/encryption.yaml +chmod 0600 /etc/kubernetes/encryption.yaml +``` + +- Set the contents to: + +``` yaml +apiVersion: v1 +kind: EncryptionConfig +resources: + - resources: + - secrets + providers: + - aescbc: + keys: + - name: key1 + secret: <32-byte base64 encoded string> + - identity: {} +``` + +Where `secret` is the 32-byte base64-encoded string generated in the first step. + +### 1.1.3 - Install the audit log configuration on all control plane nodes. + +**Profile Applicability** + +- Level 1 + +**Description** + +Place the configuration file for Kubernetes audit logging on each of the control plane nodes in the cluster. + +**Rationale** + +The Kubernetes API has audit logging capability that is the best way to track actions in the cluster. + +This supports the following controls: + +- 1.1.15 - Ensure that the `--audit-log-path` argument is set as appropriate (Scored) +- 1.1.16 - Ensure that the `--audit-log-maxage` argument is as appropriate (Scored) +- 1.1.17 - Ensure that the `--audit-log-maxbackup` argument is set as appropriate (Scored) +- 1.1.18 - Ensure that the `--audit-log-maxsize` argument is set as appropriate (Scored) +- 1.1.37 - Ensure that the `AdvancedAuditing` argument is not set to false (Scored) + +**Audit** + +On each control plane node, run: + +``` bash +stat /etc/kubernetes/audit.yaml +``` + +Ensure that: + +- The file is present +- The file mode is `0600` +- The file owner is `root:root` +- The file contains: + +``` yaml +apiVersion: audit.k8s.io/v1beta1 +kind: Policy +rules: +- level: Metadata +``` + +**Remediation** + +On nodes with the `controlplane` role: + +- Generate an empty configuration file: + +``` bash +touch /etc/kubernetes/audit.yaml +``` + +- Set the file ownership to `root:root` and the permissions to `0600` + +``` bash +chown root:root /etc/kubernetes/audit.yaml +chmod 0600 /etc/kubernetes/audit.yaml +``` + +- Set the contents to: + +``` yaml +apiVersion: audit.k8s.io/v1beta1 +kind: Policy +rules: +- level: Metadata +``` + +### 1.1.4 - Place Kubernetes event limit configuration on each control plane host + +**Profile Applicability** + +- Level 1 + +**Description** + +Place the configuration file for Kubernetes event limit configuration on each of the control plane nodes in the cluster. + +**Rationale** + +Set up the `EventRateLimit` admission control plugin to prevent clients from overwhelming the API server. The settings below are intended as an initial value and may need to be adjusted for larger clusters. + +This supports the following control: + +- 1.1.36 - Ensure that the admission control plugin `EventRateLimit` is set (Scored) + +**Audit** + +On nodes with the `controlplane` role run: + +``` bash +stat /etc/kubernetes/admission.yaml +stat /etc/kubernetes/event.yaml +``` + +For each file, ensure that: + +- The file is present +- The file mode is `0600` +- The file owner is `root:root` + +For `admission.yaml` ensure that the file contains: + +``` yaml +apiVersion: apiserver.k8s.io/v1alpha1 +kind: AdmissionConfiguration +plugins: +- name: EventRateLimit + path: /etc/kubernetes/event.yaml +``` + +For `event.yaml` ensure that the file contains: + +``` yaml +apiVersion: eventratelimit.admission.k8s.io/v1alpha1 +kind: Configuration +limits: +- type: Server + qps: 500 + burst: 5000 +``` + +**Remediation** + +On nodes with the `controlplane` role: + +- Generate an empty configuration file: + +``` bash +touch /etc/kubernetes/admission.yaml +touch /etc/kubernetes/event.yaml +``` + +- Set the file ownership to `root:root` and the permissions to `0600` + +``` bash +chown root:root /etc/kubernetes/admission.yaml +chown root:root /etc/kubernetes/event.yaml +chmod 0600 /etc/kubernetes/admission.yaml +chmod 0600 /etc/kubernetes/event.yaml +``` + +- For `admission.yaml` set the contents to: + +``` yaml +apiVersion: apiserver.k8s.io/v1alpha1 +kind: AdmissionConfiguration +plugins: +- name: EventRateLimit + path: /etc/kubernetes/event.yaml +``` + +- For event.yaml set the contents to: + +``` yaml +apiVersion: eventratelimit.admission.k8s.io/v1alpha1 +kind: Configuration +limits: +- type: Server + qps: 500 + burst: 5000 +``` + +## 2.1 - Rancher HA Kubernetes Cluster Configuration via RKE + +(See Appendix A. for full RKE `cluster.yml` example) + +### 2.1.1 - Configure kubelet options + +**Profile Applicability** + +- Level 1 + +**Description** + +Ensure Kubelet options are configured to match CIS controls. + +**Rationale** + +To pass the following controls in the CIS benchmark, ensure the appropriate flags are passed to the Kubelet. + +- 2.1.6 - Ensure that the --streaming-connection-idle-timeout argument is not set to 0 (Scored) +- 2.1.7 - Ensure that the --protect-kernel-defaults argument is set to true (Scored) +- 2.1.8 - Ensure that the --make-iptables-util-chains argument is set to true (Scored) +- 2.1.10 - Ensure that the --event-qps argument is set to 0 (Scored) + +**Audit** + +Inspect the Kubelet containers on all hosts and verify that they are running with the following options: + +- `--streaming-connection-idle-timeout=` +- `--protect-kernel-defaults=false` +- `--make-iptables-util-chains=false` +- `--event-qps=0` + +**Remediation** + +- Add the following to the RKE `cluster.yml` kubelet section under `services`: + +``` yaml +services: + kubelet: + extra_args: + streaming-connection-idle-timeout: "" + protect-kernel-defaults: "true" + make-iptables-util-chains: "true" + event-qps: "0" +``` + + Where `` is in a form like `1800s`. + +- Reconfigure the cluster: + +``` bash +rke up --config cluster.yml +``` + +### 2.1.2 - Configure kube-api options + +**Profile Applicability** + +- Level 1 + +**Description** + +Ensure the RKE configuration is set to deploy the `kube-api` service with the options required for controls. + +**Rationale** + +To pass the following controls for the kube-api server ensure RKE configuration passes the appropriate options. + +- 1.1.1 - Ensure that the `--anonymous-auth` argument is set to false (Scored) +- 1.1.8 - Ensure that the `--profiling argument` is set to false (Scored) +- 1.1.9 - Ensure that the `--repair-malformed-updates` argument is set to false (Scored) +- 1.1.11 - Ensure that the admission control plugin `AlwaysPullImages` is set (Scored) +- 1.1.12 - Ensure that the admission control plugin `DenyEscalatingExec` is set (Scored) +- 1.1.14 - Ensure that the admission control plugin `NamespaceLifecycle` is set (Scored) +- 1.1.15 - Ensure that the `--audit-log-path` argument is set as appropriate (Scored) +- 1.1.16 - Ensure that the `--audit-log-maxage` argument is set as appropriate (Scored) +- 1.1.17 - Ensure that the `--audit-log-maxbackup` argument is set as appropriate (Scored) +- 1.1.18 - Ensure that the `--audit-log-maxsize` argument is set as appropriate (Scored) +- 1.1.23 - Ensure that the `--service-account-lookup` argument is set to true (Scored) +- 1.1.24 - Ensure that the admission control plugin `PodSecurityPolicy` is set (Scored) +- 1.1.34 - Ensure that the `--experimental-encryption-provider-config` argument is set as appropriate (Scored) +- 1.1.35 - Ensure that the encryption provider is set to `aescbc` (Scored) +- 1.1.36 - Ensure that the admission control plugin `EventRateLimit` is set (Scored) +- 1.1.37 - Ensure that the `AdvancedAuditing` argument is not set to `false` (Scored) + +**Audit** + +- On nodes with the `controlplane` role inspect the `kube-apiserver` containers: + + ``` bash + docker inspect kube-apiserver + ``` + +- Look for the following options in the command section of the output: + +``` text +--anonymous-auth=false +--profiling=false +--repair-malformed-updates=false +--service-account-lookup=true +--enable-admission-plugins= "ServiceAccount,NamespaceLifecycle,LimitRanger,PersistentVolumeLabel,DefaultStorageClass,ResourceQuota,DefaultTolerationSeconds,AlwaysPullImages,DenyEscalatingExec,NodeRestriction,EventRateLimit,PodSecurityPolicy" +--experimental-encryption-provider-config=/etc/kubernetes/encryption.yaml +--admission-control-config-file=/etc/kubernetes/admission.yaml +--audit-log-path=/var/log/kube-audit/audit-log.json +--audit-log-maxage=5 +--audit-log-maxbackup=5 +--audit-log-maxsize=100 +--audit-log-format=json +--audit-policy-file=/etc/kubernetes/audit.yaml +``` + +- In the `volume` section of the output ensure the bind mount is present: + +``` text +/var/log/kube-audit:/var/log/kube-audit +``` + +**Remediation** + +- In the RKE `cluster.yml` add the following directives to the `kube-api` section under `services`: + +``` yaml +services: + kube-api: + pod_security_policy: true + extra_args: + anonymous-auth: "false" + profiling: "false" + repair-malformed-updates: "false" + service-account-lookup: "true" + enable-admission-plugins: "ServiceAccount,NamespaceLifecycle,LimitRanger,PersistentVolumeLabel,DefaultStorageClass,ResourceQuota,DefaultTolerationSeconds,AlwaysPullImages,DenyEscalatingExec,NodeRestriction,EventRateLimit,PodSecurityPolicy" + experimental-encryption-provider-config: /etc/kubernetes/encryption.yaml + admission-control-config-file: "/etc/kubernetes/admission.yaml" + audit-log-path: "/var/log/kube-audit/audit-log.json" + audit-log-maxage: "5" + audit-log-maxbackup: "5" + audit-log-maxsize: "100" + audit-log-format: "json" + audit-policy-file: /etc/kubernetes/audit.yaml + extra_binds: + - "/var/log/kube-audit:/var/log/kube-audit" +``` + +- Reconfigure the cluster: + +``` bash +rke up --config cluster.yml +``` + +### 2.1.3 - Configure scheduler options + +**Profile Applicability** + +- Level 1 + +**Description** + +Set the appropriate options for the Kubernetes scheduling service. + +**Rationale** + +To address the following controls on the CIS benchmark, the command line options should be set on the Kubernetes scheduler. + +- 1.2.1 - Ensure that the `--profiling` argument is set to `false` (Scored) +- 1.2.2 - Ensure that the `--address` argument is set to `127.0.0.1` (Scored) + +**Audit** + +- On nodes with the `controlplane` role: inspect the `kube-scheduler` containers: + +``` bash +docker inspect kube-scheduler +``` + +- Verify the following options are set in the `command` section. + +``` text +--profiling=false +--address=127.0.0.1 +``` + +**Remediation** + +- In the RKE `cluster.yml` file ensure the following options are set: + +``` yaml +services: + … + scheduler: + extra_args: + profiling: "false" + address: "127.0.0.1" +``` + +- Reconfigure the cluster: + +``` bash +rke up --config cluster.yml +``` + +### 2.1.4 - Configure controller options + +**Profile Applicability** + +- Level 1 + +**Description** + +Set the appropriate arguments on the Kubernetes controller manager. + +**Rationale** + +To address the following controls the options need to be passed to the Kubernetes controller manager. + +- 1.3.1 - Ensure that the --terminated-pod-gc-threshold argument is set as appropriate (Scored) +- 1.3.2 - Ensure that the --profiling argument is set to false (Scored) +- 1.3.7 - Ensure that the --address argument is set to 127.0.0.1 (Scored) + +**Audit** + +- On nodes with the `controlplane` role inspect the `kube-controller-manager` container: + +``` bash +docker inspect kube-controller-manager +``` + +- Verify the following options are set in the `command` section: + +``` text +--terminated-pod-gc-threshold=1000 +--profiling=false +--address=127.0.0.1 +``` + +**Remediation** + +- In the RKE cluster.yml file ensure the following options are set: + +``` yaml +services: + kube-controller: + extra_args: + profiling: "false" + address: "127.0.0.1" + terminated-pod-gc-threshold: "1000" +``` + +- Reconfigure the cluster: + +``` bash +rke up --config cluster.yml +``` + +### 2.1.5 - Configure addons and PSPs + +**Profile Applicability** + +- Level 1 + +**Description** + +Configure a restrictive pod security policy (PSP) as the default and create role bindings for system level services to use the less restrictive default PSP. + +**Rationale** + +To address the following controls, a restrictive default PSP needs to be applied as the default. Role bindings need to be in place to allow system services to still function. + +- 1.7.1 - Do not admit privileged containers (Not Scored) +- 1.7.2 - Do not admit containers wishing to share the host process ID namespace (Not Scored) +- 1.7.3 - Do not admit containers wishing to share the host IPC namespace (Not Scored) +- 1.7.4 - Do not admit containers wishing to share the host network namespace (Not Scored) +- 1.7.5 - Do not admit containers with `allowPrivilegeEscalation` (Not Scored) +- 1.7.6 - Do not admit root containers (Not Scored) +- 1.7.7 - Do not admit containers with dangerous capabilities (Not Scored) + +**Audit** + +- Verify that the `cattle-system` namespace exists: + +``` bash +kubectl get ns |grep cattle +``` + +- Verify that the roles exist: + +``` bash +kubectl get role default-psp-role -n ingress-nginx +kubectl get role default-psp-role -n cattle-system +kubectl get clusterrole psp:restricted +``` + +- Verify the bindings are set correctly: + +``` bash +kubectl get rolebinding -n ingress-nginx default-psp-rolebinding +kubectl get rolebinding -n cattle-system default-psp-rolebinding +kubectl get clusterrolebinding psp:restricted +``` + +- Verify the restricted PSP is present. + +``` bash +kubectl get psp restricted +``` + +**Remediation** + +- In the RKE `cluster.yml` file ensure the following options are set: + +``` yaml +addons: | + apiVersion: rbac.authorization.k8s.io/v1 + kind: Role + metadata: + name: default-psp-role + namespace: ingress-nginx + rules: + - apiGroups: + - extensions + resourceNames: + - default-psp + resources: + - podsecuritypolicies + verbs: + - use + --- + apiVersion: rbac.authorization.k8s.io/v1 + kind: RoleBinding + metadata: + name: default-psp-rolebinding + namespace: ingress-nginx + roleRef: + apiGroup: rbac.authorization.k8s.io + kind: Role + name: default-psp-role + subjects: + - apiGroup: rbac.authorization.k8s.io + kind: Group + name: system:serviceaccounts + - apiGroup: rbac.authorization.k8s.io + kind: Group + name: system:authenticated + --- + apiVersion: v1 + kind: Namespace + metadata: + name: cattle-system + --- + apiVersion: rbac.authorization.k8s.io/v1 + kind: Role + metadata: + name: default-psp-role + namespace: cattle-system + rules: + - apiGroups: + - extensions + resourceNames: + - default-psp + resources: + - podsecuritypolicies + verbs: + - use + --- + apiVersion: rbac.authorization.k8s.io/v1 + kind: RoleBinding + metadata: + name: default-psp-rolebinding + namespace: cattle-system + roleRef: + apiGroup: rbac.authorization.k8s.io + kind: Role + name: default-psp-role + subjects: + - apiGroup: rbac.authorization.k8s.io + kind: Group + name: system:serviceaccounts + - apiGroup: rbac.authorization.k8s.io + kind: Group + name: system:authenticated + --- + apiVersion: extensions/v1beta1 + kind: PodSecurityPolicy + metadata: + name: restricted + spec: + requiredDropCapabilities: + - NET_RAW + privileged: false + allowPrivilegeEscalation: false + defaultAllowPrivilegeEscalation: false + fsGroup: + rule: RunAsAny + runAsUser: + rule: MustRunAsNonRoot + seLinux: + rule: RunAsAny + supplementalGroups: + rule: RunAsAny + volumes: + - emptyDir + - secret + - persistentVolumeClaim + - downwardAPI + - configMap + - projected + --- + apiVersion: rbac.authorization.k8s.io/v1 + kind: ClusterRole + metadata: + name: psp:restricted + rules: + - apiGroups: + - extensions + resourceNames: + - restricted + resources: + - podsecuritypolicies + verbs: + - use + --- + apiVersion: rbac.authorization.k8s.io/v1 + kind: ClusterRoleBinding + metadata: + name: psp:restricted + roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: psp:restricted + subjects: + - apiGroup: rbac.authorization.k8s.io + kind: Group + name: system:serviceaccounts + - apiGroup: rbac.authorization.k8s.io + kind: Group + name: system:authenticated +``` + +- Reconfigure the cluster: + +``` bash +rke up --config cluster.yml +``` + +## 3.1 - Rancher Management Control Plane Installation + +### 3.1.1 - Disable the local cluster option + +**Profile Applicability** + +- Level 2 + +**Description** + +When deploying Rancher, disable the local cluster option on the Rancher Server. + +**NOTE:** This requires Rancher v2.1.2 or above. + +**Rationale** + +Having access to the local cluster from the Rancher UI is convenient for troubleshooting and debugging; however, if the local cluster is enabled in the Rancher UI, a user has access to all elements of the system, including the Rancher management server itself. Disabling the local cluster is a defense in depth measure and removes the possible attack vector from the Rancher UI and API. + +**Audit** + +- Verify the Rancher deployment has the `--add-local=false` option set. + +``` bash +kubectl get deployment rancher -n cattle-system -o yaml |grep 'add-local' +``` + +- In the Rancher UI go to _Clusters_ in the _Global_ view and verify that no `local` cluster is present. On a fresh install the _Clusters_ tab will look like the following image: + + ![No Local Cluster](images/no-local-cluster.png) + +**Remediation** + +- Upgrade to Rancher v2.1.2 via the Helm chart. While performing the upgrade, provide the following installation flag: + +``` text +--set addLocal="false" +``` + +### 3.1.2 - Enable Rancher Audit logging + +**Profile Applicability** + +- Level 1 + +**Description** + +Enable Rancher’s built-in audit logging capability. + +**Rationale** + +Tracking down what actions were performed by users in Rancher can provide insight during post mortems, and if monitored proactively can be used to quickly detect malicious actions. + +**Audit** + +- Verify that the audit log parameters were passed into the Rancher deployment. + +``` +kubectl get deployment rancher -n cattle-system -o yaml | grep auditLog +``` + +- Verify that the log is going to the appropriate destination, as set by +`auditLog.destination` + + - `sidecar`: + + 1. List pods: + + ``` bash + kubectl get pods -n cattle-system + ``` + + 2. Tail logs: + + ``` bash + kubectl logs -n cattle-system -c rancher-audit-log + ``` + + - `hostPath` + + 1. On the worker nodes running the Rancher pods, verify that the log files are being written to the destination indicated in `auditlog.hostPath`. + +**Remediation** + +Upgrade the Rancher server installation using Helm, and configure the audit log settings. The instructions for doing so can be found in the reference section below. + +#### Reference + +- + +## 3.2 - Rancher Management Control Plane Authentication + +### 3.2.1 - Change the local admin password from the default value + +**Profile Applicability** + +- Level 1 + +**Description** + +The local admin password should be changed from the default. + +**Rationale** + +The default admin password is common across all Rancher installations and should be changed immediately upon startup. + +**Audit** + +Attempt to login into the UI with the following credentials: + - Username: admin + - Password: admin + +The login attempt must not succeed. + +**Remediation** + +Change the password from `admin` to a password that meets the recommended password standards for your organization. + +### 3.2.2 - Configure an Identity Provider for Authentication + +**Profile Applicability** + +- Level 1 + +**Description** + +When running Rancher in a production environment, configure an identity provider for authentication. + +**Rationale** + +Rancher supports several authentication backends that are common in enterprises. It is recommended to tie Rancher into an external authentication system to simplify user and group access in the Rancher cluster. Doing so assures that access control follows the organization's change management process for user accounts. + +**Audit** + +- In the Rancher UI, select _Global_ +- Select _Security_ +- Select _Authentication_ +- Ensure the authentication provider for your environment is active and configured correctly + +**Remediation** + +Configure the appropriate authentication provider for your Rancher installation according to the documentation found at the link in the reference section below. + +#### Reference + +- + +## 3.3 - Rancher Management Control Plane RBAC + +### 3.3.1 - Ensure that administrator privileges are only granted to those who require them + +**Profile Applicability** + +- Level 1 + +**Description** + +Restrict administrator access to only those responsible for managing and operating the Rancher server. + +**Rationale** + +The `admin` privilege level gives the user the highest level of access to the Rancher server and all attached clusters. This privilege should only be granted to a few people who are responsible for the availability and support of Rancher and the clusters that it manages. + +**Audit** + +The following script uses the Rancher API to show users with administrator privileges: + +``` bash +#!/bin/bash +for i in $(curl -sk -u 'token-:' https:///v3/users|jq -r .data[].links.globalRoleBindings); do + +curl -sk -u 'token-:' $i| jq '.data[] | "\(.userId) \(.globalRoleId)"' + +done + +``` + +The `admin` role should only be assigned to users that require administrative privileges. Any role that is not `admin` or `user` should be audited in the RBAC section of the UI to ensure that the privileges adhere to policies for global access. + +The Rancher server permits customization of the default global permissions. We recommend that auditors also review the policies of any custom global roles. + +**Remediation** + +Remove the `admin` role from any user that does not require administrative privileges. + +## 3.4 - Rancher Management Control Plane Configuration + +### 3.4.1 - Ensure only approved node drivers are active + +**Profile Applicability** + +- Level 1 + +**Description** + +Ensure that node drivers that are not needed or approved are not active in the Rancher console. + +**Rationale** + +Node drivers are used to provision compute nodes in various cloud providers and local IaaS infrastructure. For convenience, popular cloud providers are enabled by default. If the organization does not intend to use these or does not allow users to provision resources in certain providers, the drivers should be disabled. This will prevent users from using Rancher resources to provision the nodes. + +**Audit** + +- In the Rancher UI select _Global_ +- Select _Node Drivers_ +- Review the list of node drivers that are in an _Active_ state. + +**Remediation** + +If a disallowed node driver is active, visit the _Node Drivers_ page under _Global_ and disable it. + +--- + +## Appendix A - Complete RKE `cluster.yml` Example + +``` yaml +nodes: +- address: 18.191.190.205 + internal_address: 172.31.24.213 + user: ubuntu + role: [ "controlplane", "etcd", "worker" ] +- address: 18.191.190.203 + internal_address: 172.31.24.203 + user: ubuntu + role: [ "controlplane", "etcd", "worker" ] +- address: 18.191.190.10 + internal_address: 172.31.24.244 + user: ubuntu + role: [ "controlplane", "etcd", "worker" ] + +services: + kubelet: + extra_args: + streaming-connection-idle-timeout: "1800s" + protect-kernel-defaults: "true" + make-iptables-util-chains: "true" + event-qps: "0" + kube-api: + pod_security_policy: true + extra_args: + anonymous-auth: "false" + profiling: "false" + repair-malformed-updates: "false" + service-account-lookup: "true" + enable-admission-plugins: "ServiceAccount,NamespaceLifecycle,LimitRanger,PersistentVolumeLabel,DefaultStorageClass,ResourceQuota,DefaultTolerationSeconds,AlwaysPullImages,DenyEscalatingExec,NodeRestriction,EventRateLimit,PodSecurityPolicy" + experimental-encryption-provider-config: /etc/kubernetes/encryption.yaml + admission-control-config-file: "/etc/kubernetes/admission.yaml" + audit-log-path: "/var/log/kube-audit/audit-log.json" + audit-log-maxage: "5" + audit-log-maxbackup: "5" + audit-log-maxsize: "100" + audit-log-format: "json" + audit-policy-file: /etc/kubernetes/audit.yaml + extra_binds: + - "/var/log/kube-audit:/var/log/kube-audit" + scheduler: + extra_args: + profiling: "false" + address: "127.0.0.1" + kube-controller: + extra_args: + profiling: "false" + address: "127.0.0.1" + terminated-pod-gc-threshold: "1000" +addons: | + apiVersion: rbac.authorization.k8s.io/v1 + kind: Role + metadata: + name: default-psp-role + namespace: ingress-nginx + rules: + - apiGroups: + - extensions + resourceNames: + - default-psp + resources: + - podsecuritypolicies + verbs: + - use + --- + apiVersion: rbac.authorization.k8s.io/v1 + kind: RoleBinding + metadata: + name: default-psp-rolebinding + namespace: ingress-nginx + roleRef: + apiGroup: rbac.authorization.k8s.io + kind: Role + name: default-psp-role + subjects: + - apiGroup: rbac.authorization.k8s.io + kind: Group + name: system:serviceaccounts + - apiGroup: rbac.authorization.k8s.io + kind: Group + name: system:authenticated + --- + apiVersion: v1 + kind: Namespace + metadata: + name: cattle-system + --- + apiVersion: rbac.authorization.k8s.io/v1 + kind: Role + metadata: + name: default-psp-role + namespace: cattle-system + rules: + - apiGroups: + - extensions + resourceNames: + - default-psp + resources: + - podsecuritypolicies + verbs: + - use + --- + apiVersion: rbac.authorization.k8s.io/v1 + kind: RoleBinding + metadata: + name: default-psp-rolebinding + namespace: cattle-system + roleRef: + apiGroup: rbac.authorization.k8s.io + kind: Role + name: default-psp-role + subjects: + - apiGroup: rbac.authorization.k8s.io + kind: Group + name: system:serviceaccounts + - apiGroup: rbac.authorization.k8s.io + kind: Group + name: system:authenticated + --- + apiVersion: extensions/v1beta1 + kind: PodSecurityPolicy + metadata: + name: restricted + spec: + requiredDropCapabilities: + - NET_RAW + privileged: false + allowPrivilegeEscalation: false + defaultAllowPrivilegeEscalation: false + fsGroup: + rule: RunAsAny + runAsUser: + rule: MustRunAsNonRoot + seLinux: + rule: RunAsAny + supplementalGroups: + rule: RunAsAny + volumes: + - emptyDir + - secret + - persistentVolumeClaim + - downwardAPI + - configMap + - projected + --- + apiVersion: rbac.authorization.k8s.io/v1 + kind: ClusterRole + metadata: + name: psp:restricted + rules: + - apiGroups: + - extensions + resourceNames: + - restricted + resources: + - podsecuritypolicies + verbs: + - use + --- + apiVersion: rbac.authorization.k8s.io/v1 + kind: ClusterRoleBinding + metadata: + name: psp:restricted + roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: psp:restricted + subjects: + - apiGroup: rbac.authorization.k8s.io + kind: Group + name: system:serviceaccounts + - apiGroup: rbac.authorization.k8s.io + kind: Group + name: system:authenticated +``` \ No newline at end of file