diff --git a/content/rancher/v2.x/en/security/rancher-2.5/1.6-hardening-2.5/_index.md b/content/rancher/v2.x/en/security/rancher-2.5/1.6-hardening-2.5/_index.md index 599b75bf495..efc9c393e69 100644 --- a/content/rancher/v2.x/en/security/rancher-2.5/1.6-hardening-2.5/_index.md +++ b/content/rancher/v2.x/en/security/rancher-2.5/1.6-hardening-2.5/_index.md @@ -1,9 +1,9 @@ --- -title: Hardening Guide with CIS 1.5 Benchmark -weight: 99 +title: Hardening Guide with CIS 1.6 Benchmark +weight: 100 --- -This document provides prescriptive guidance for hardening a production installation of a RKE cluster to be used with Rancher v2.5. It outlines the configurations and controls required to address Kubernetes benchmark controls from the Center for Information Security (CIS). +This document provides prescriptive guidance for hardening a production installation of a RKE cluster to be used with Rancher v2.5.4. It outlines the configurations and controls required to address Kubernetes benchmark controls from the Center for Information Security (CIS). > This hardening guide describes how to secure the nodes in your cluster, and it is recommended to follow this guide before installing Kubernetes. @@ -11,26 +11,26 @@ This hardening guide is intended to be used for RKE clusters and associated with Rancher Version | CIS Benchmark Version | Kubernetes Version ----------------|-----------------------|------------------ - Rancher v2.5 | Benchmark v1.5 | Kubernetes 1.15 + Rancher v2.5.4 | Benchmark 1.6 | Kubernetes v1.18 -[Click here to download a PDF version of this document](https://releases.rancher.com/documents/security/2.5/Rancher_Hardening_Guide_CIS_1.5.pdf) +[Click here to download a PDF version of this document](https://releases.rancher.com/documents/security/2.5/Rancher_Hardening_Guide_CIS_1.6.pdf) ### Overview -This document provides prescriptive guidance for hardening a RKE cluster to be used for installing Rancher v2.5 with Kubernetes v1.15 or provisioning a RKE cluster with Kubernetes 1.15 to be used within Rancher v2.5. It outlines the configurations required to address Kubernetes benchmark controls from the Center for Information Security (CIS). +This document provides prescriptive guidance for hardening a RKE cluster to be used for installing Rancher v2.5.4 with Kubernetes v1.18 or provisioning a RKE cluster with Kubernetes v1.18 to be used within Rancher v2.5.4. It outlines the configurations required to address Kubernetes benchmark controls from the Center for Information Security (CIS). -For more detail about evaluating a hardened cluster against the official CIS benchmark, refer to the [CIS 1.5 Benchmark - Self-Assessment Guide - Rancher v2.5]({{< baseurl >}}/rancher/v2.x/en/security/rancher-2.5/1.5-benchmark-2.5/). +For more detail about evaluating a hardened cluster against the official CIS benchmark, refer to the [CIS 1.6 Benchmark - Self-Assessment Guide - Rancher v2.5.4]({{< baseurl >}}/rancher/v2.x/en/security/rancher-2.5/1.6-benchmark-2.5/). #### Known Issues -- Rancher **exec shell** and **view logs** for pods are **not** functional in a CIS 1.5 hardened setup when only public IP is provided when registering custom nodes. This functionality requires a private IP to be provided when registering the custom nodes. -- When setting the `default_pod_security_policy_template_id:` to `restricted` Rancher creates **RoleBindings** and **ClusterRoleBindings** on the default service accounts. The CIS 1.5 5.1.5 check requires the default service accounts have no roles or cluster roles bound to it apart from the defaults. In addition the default service accounts should be configured such that it does not provide a service account token and does not have any explicit rights assignments. +- Rancher **exec shell** and **view logs** for pods are **not** functional in a CIS 1.6 hardened setup when only public IP is provided when registering custom nodes. This functionality requires a private IP to be provided when registering the custom nodes. +- When setting the `default_pod_security_policy_template_id:` to `restricted` Rancher creates **RoleBindings** and **ClusterRoleBindings** on the default service accounts. The CIS 1.6 5.1.5 check requires the default service accounts have no roles or cluster roles bound to it apart from the defaults. In addition the default service accounts should be configured such that it does not provide a service account token and does not have any explicit rights assignments. ### Configure Kernel Runtime Parameters The following `sysctl` configuration is recommended for all nodes type in the cluster. Set the following parameters in `/etc/sysctl.d/90-kubelet.conf`: -``` +```ini vm.overcommit_memory=1 vm.panic_on_oom=0 kernel.panic=10 @@ -48,14 +48,14 @@ To create the **etcd** group run the following console commands. The commands below use `52034` for **uid** and **gid** are for example purposes. Any valid unused **uid** or **gid** could also be used in lieu of `52034`. -``` +```bash groupadd --gid 52034 etcd useradd --comment "etcd service account" --uid 52034 --gid 52034 etcd ``` Update the RKE **config.yml** with the **uid** and **gid** of the **etcd** user: -``` yaml +```yaml services: etcd: gid: 52034 @@ -67,13 +67,13 @@ Kubernetes provides a default service account which is used by cluster workloads For each namespace including **default** and **kube-system** on a standard RKE install the **default** service account must include this value: -``` +```yaml automountServiceAccountToken: false ``` Save the following yaml to a file called `account_update.yaml` -``` yaml +```yaml apiVersion: v1 kind: ServiceAccount metadata: @@ -83,7 +83,7 @@ automountServiceAccountToken: false Create a bash script file called `account_update.sh`. Be sure to `chmod +x account_update.sh` so the script has execute permissions. -``` +```bash #!/bin/bash -e for namespace in $(kubectl get namespaces -A -o json | jq -r '.items[].metadata.name'); do @@ -116,7 +116,7 @@ about network policies can be found on the Kubernetes site. > This `NetworkPolicy` is not recommended for production use -``` yaml +```yaml --- apiVersion: networking.k8s.io/v1 kind: NetworkPolicy @@ -136,13 +136,14 @@ spec: Create a bash script file called `apply_networkPolicy_to_all_ns.sh`. Be sure to `chmod +x apply_networkPolicy_to_all_ns.sh` so the script has execute permissions. -``` +```bash #!/bin/bash -e for namespace in $(kubectl get namespaces -A -o json | jq -r '.items[].metadata.name'); do kubectl apply -f default-allow-all.yaml -n ${namespace} done ``` + Execute this script to apply the `default-allow-all.yaml` the **permissive** `NetworkPolicy` to all namespaces. ### Reference Hardened RKE `cluster.yml` configuration @@ -152,139 +153,111 @@ of Rancher Kubernetes Engine (RKE). Install [documentation](https://rancher.com/ provided with additional details about the configuration items. This reference `cluster.yml` does not include the required **nodes** directive which will vary depending on your environment. Documentation for node configuration can be found here: https://rancher.com/docs/rke/latest/en/config-options/nodes -``` yaml +```yaml # If you intend to deploy Kubernetes in an air-gapped environment, # please consult the documentation on how to configure custom RKE images. -kubernetes_version: "v1.15.9-rancher1-1" -enable_network_policy: true -default_pod_security_policy_template_id: "restricted" +# https://rancher.com/docs/rke/latest/en/installation/ + # the nodes directive is required and will vary depending on your environment # documentation for node configuration can be found here: -# https://rancher.com/docs/rke/latest/en/config-options/nodes -nodes: +# https://rancher.com/docs/rke/latest/en/config-options/nodes +nodes: [] services: etcd: + image: "" + extra_args: {} + extra_binds: [] + extra_env: [] + win_extra_args: {} + win_extra_binds: [] + win_extra_env: [] + external_urls: [] + ca_cert: "" + cert: "" + key: "" + path: "" uid: 52034 gid: 52034 + snapshot: true + retention: "" + creation: "" + backup_config: null kube-api: + image: "" + extra_args: {} + extra_binds: [] + extra_env: [] + win_extra_args: {} + win_extra_binds: [] + win_extra_env: [] + service_cluster_ip_range: "" + service_node_port_range: "" pod_security_policy: true + always_pull_images: false secrets_encryption_config: enabled: true + custom_config: null audit_log: enabled: true - admission_configuration: + configuration: null + admission_configuration: null event_rate_limit: enabled: true + configuration: null kube-controller: + image: "" extra_args: - feature-gates: "RotateKubeletServerCertificate=true" + feature-gates: RotateKubeletServerCertificate=true + extra_binds: [] + extra_env: [] + win_extra_args: {} + win_extra_binds: [] + win_extra_env: [] + cluster_cidr: "" + service_cluster_ip_range: "" scheduler: image: "" extra_args: {} extra_binds: [] extra_env: [] + win_extra_args: {} + win_extra_binds: [] + win_extra_env: [] kubelet: - generate_serving_certificate: true + image: "" extra_args: - feature-gates: "RotateKubeletServerCertificate=true" + feature-gates: RotateKubeletServerCertificate=true protect-kernel-defaults: "true" - tls-cipher-suites: "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256" + tls-cipher-suites: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 extra_binds: [] extra_env: [] - cluster_domain: "" + win_extra_args: {} + win_extra_binds: [] + win_extra_env: [] + cluster_domain: cluster.local infra_container_image: "" cluster_dns_server: "" fail_swap_on: false + generate_serving_certificate: true kubeproxy: image: "" extra_args: {} extra_binds: [] extra_env: [] + win_extra_args: {} + win_extra_binds: [] + win_extra_env: [] network: plugin: "" options: {} mtu: 0 node_selector: {} + update_strategy: null authentication: strategy: "" sans: [] webhook: null addons: | - --- - apiVersion: v1 - kind: Namespace - metadata: - name: ingress-nginx - --- - apiVersion: rbac.authorization.k8s.io/v1 - kind: Role - metadata: - name: default-psp-role - namespace: ingress-nginx - rules: - - apiGroups: - - extensions - resourceNames: - - default-psp - resources: - - podsecuritypolicies - verbs: - - use - --- - apiVersion: rbac.authorization.k8s.io/v1 - kind: RoleBinding - metadata: - name: default-psp-rolebinding - namespace: ingress-nginx - roleRef: - apiGroup: rbac.authorization.k8s.io - kind: Role - name: default-psp-role - subjects: - - apiGroup: rbac.authorization.k8s.io - kind: Group - name: system:serviceaccounts - - apiGroup: rbac.authorization.k8s.io - kind: Group - name: system:authenticated - --- - apiVersion: v1 - kind: Namespace - metadata: - name: cattle-system - --- - apiVersion: rbac.authorization.k8s.io/v1 - kind: Role - metadata: - name: default-psp-role - namespace: cattle-system - rules: - - apiGroups: - - extensions - resourceNames: - - default-psp - resources: - - podsecuritypolicies - verbs: - - use - --- - apiVersion: rbac.authorization.k8s.io/v1 - kind: RoleBinding - metadata: - name: default-psp-rolebinding - namespace: cattle-system - roleRef: - apiGroup: rbac.authorization.k8s.io - kind: Role - name: default-psp-role - subjects: - - apiGroup: rbac.authorization.k8s.io - kind: Group - name: system:serviceaccounts - - apiGroup: rbac.authorization.k8s.io - kind: Group - name: system:authenticated - --- apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: @@ -311,55 +284,25 @@ addons: | - configMap - projected --- - apiVersion: rbac.authorization.k8s.io/v1 - kind: ClusterRole + apiVersion: networking.k8s.io/v1 + kind: NetworkPolicy metadata: - name: psp:restricted - rules: - - apiGroups: - - extensions - resourceNames: - - restricted - resources: - - podsecuritypolicies - verbs: - - use - --- - apiVersion: rbac.authorization.k8s.io/v1 - kind: ClusterRoleBinding - metadata: - name: psp:restricted - roleRef: - apiGroup: rbac.authorization.k8s.io - kind: ClusterRole - name: psp:restricted - subjects: - - apiGroup: rbac.authorization.k8s.io - kind: Group - name: system:serviceaccounts - - apiGroup: rbac.authorization.k8s.io - kind: Group - name: system:authenticated + name: default-allow-all + spec: + podSelector: {} + ingress: + - {} + egress: + - {} + policyTypes: + - Ingress + - Egress --- apiVersion: v1 kind: ServiceAccount metadata: - name: tiller - namespace: kube-system - --- - apiVersion: rbac.authorization.k8s.io/v1 - kind: ClusterRoleBinding - metadata: - name: tiller - roleRef: - apiGroup: rbac.authorization.k8s.io - kind: ClusterRole - name: cluster-admin - subjects: - - kind: ServiceAccount - name: tiller - namespace: kube-system - + name: default + automountServiceAccountToken: false addons_include: [] system_images: etcd: "" @@ -373,6 +316,7 @@ system_images: kubedns_autoscaler: "" coredns: "" coredns_autoscaler: "" + nodelocal: "" kubernetes: "" flannel: "" flannel_cni: "" @@ -383,6 +327,7 @@ system_images: calico_flexvol: "" canal_node: "" canal_cni: "" + canal_controllers: "" canal_flannel: "" canal_flexvol: "" weave_node: "" @@ -399,6 +344,7 @@ authorization: mode: "" options: {} ignore_docker_version: false +kubernetes_version: v1.18.12-rancher1-1 private_registries: [] ingress: provider: "" @@ -409,8 +355,15 @@ ingress: extra_envs: [] extra_volumes: [] extra_volume_mounts: [] -cluster_name: "" + update_strategy: null + http_port: 0 + https_port: 0 + network_mode: "" +cluster_name: +cloud_provider: + name: "" prefix_path: "" +win_prefix_path: "" addon_job_timeout: 0 bastion_host: address: "" @@ -424,10 +377,17 @@ monitoring: provider: "" options: {} node_selector: {} + update_strategy: null + replicas: null restore: restore: false snapshot_name: "" dns: null +upgrade_strategy: + max_unavailable_worker: "" + max_unavailable_controlplane: "" + drain: null + node_drain_input: null ``` ### Reference Hardened RKE Template configuration @@ -436,198 +396,49 @@ The reference RKE Template provides the configuration needed to achieve a harden RKE Templates are used to provision Kubernetes and define Rancher settings. Follow the Rancher [documentaion](https://rancher.com/docs/rancher/v2.x/en/installation) for additional installation and RKE Template details. -``` yaml -# +```yaml +# # Cluster Config -# +# default_pod_security_policy_template_id: restricted docker_root_dir: /var/lib/docker enable_cluster_alerting: false enable_cluster_monitoring: false enable_network_policy: true -# +# # Rancher Config -# +# rancher_kubernetes_engine_config: - addon_job_timeout: 30 - addons: |- - --- - apiVersion: v1 - kind: Namespace - metadata: - name: ingress-nginx - --- - apiVersion: rbac.authorization.k8s.io/v1 - kind: Role - metadata: - name: default-psp-role - namespace: ingress-nginx - rules: - - apiGroups: - - extensions - resourceNames: - - default-psp - resources: - - podsecuritypolicies - verbs: - - use - --- - apiVersion: rbac.authorization.k8s.io/v1 - kind: RoleBinding - metadata: - name: default-psp-rolebinding - namespace: ingress-nginx - roleRef: - apiGroup: rbac.authorization.k8s.io - kind: Role - name: default-psp-role - subjects: - - apiGroup: rbac.authorization.k8s.io - kind: Group - name: system:serviceaccounts - - apiGroup: rbac.authorization.k8s.io - kind: Group - name: system:authenticated - --- - apiVersion: v1 - kind: Namespace - metadata: - name: cattle-system - --- - apiVersion: rbac.authorization.k8s.io/v1 - kind: Role - metadata: - name: default-psp-role - namespace: cattle-system - rules: - - apiGroups: - - extensions - resourceNames: - - default-psp - resources: - - podsecuritypolicies - verbs: - - use - --- - apiVersion: rbac.authorization.k8s.io/v1 - kind: RoleBinding - metadata: - name: default-psp-rolebinding - namespace: cattle-system - roleRef: - apiGroup: rbac.authorization.k8s.io - kind: Role - name: default-psp-role - subjects: - - apiGroup: rbac.authorization.k8s.io - kind: Group - name: system:serviceaccounts - - apiGroup: rbac.authorization.k8s.io - kind: Group - name: system:authenticated - --- - apiVersion: policy/v1beta1 - kind: PodSecurityPolicy - metadata: - name: restricted - spec: - requiredDropCapabilities: - - NET_RAW - privileged: false - allowPrivilegeEscalation: false - defaultAllowPrivilegeEscalation: false - fsGroup: - rule: RunAsAny - runAsUser: - rule: MustRunAsNonRoot - seLinux: - rule: RunAsAny - supplementalGroups: - rule: RunAsAny - volumes: - - emptyDir - - secret - - persistentVolumeClaim - - downwardAPI - - configMap - - projected - --- - apiVersion: rbac.authorization.k8s.io/v1 - kind: ClusterRole - metadata: - name: psp:restricted - rules: - - apiGroups: - - extensions - resourceNames: - - restricted - resources: - - podsecuritypolicies - verbs: - - use - --- - apiVersion: rbac.authorization.k8s.io/v1 - kind: ClusterRoleBinding - metadata: - name: psp:restricted - roleRef: - apiGroup: rbac.authorization.k8s.io - kind: ClusterRole - name: psp:restricted - subjects: - - apiGroup: rbac.authorization.k8s.io - kind: Group - name: system:serviceaccounts - - apiGroup: rbac.authorization.k8s.io - kind: Group - name: system:authenticated - --- - apiVersion: v1 - kind: ServiceAccount - metadata: - name: tiller - namespace: kube-system - --- - apiVersion: rbac.authorization.k8s.io/v1 - kind: ClusterRoleBinding - metadata: - name: tiller - roleRef: - apiGroup: rbac.authorization.k8s.io - kind: ClusterRole - name: cluster-admin - subjects: - - kind: ServiceAccount - name: tiller - namespace: kube-system + addon_job_timeout: 45 ignore_docker_version: true - kubernetes_version: v1.15.9-rancher1-1 -# + kubernetes_version: v1.18.12-rancher1-1 +# # If you are using calico on AWS -# +# # network: # plugin: calico # calico_network_provider: # cloud_provider: aws -# +# # # To specify flannel interface -# +# # network: # plugin: flannel # flannel_network_provider: # iface: eth1 -# +# # # To specify flannel interface for canal plugin -# +# # network: # plugin: canal # canal_network_provider: # iface: eth1 -# +# network: mtu: 0 plugin: canal -# + rotate_encryption_key: false +# # services: # kube-api: # service_cluster_ip_range: 10.43.0.0/16 @@ -637,7 +448,7 @@ rancher_kubernetes_engine_config: # kubelet: # cluster_domain: cluster.local # cluster_dns_server: 10.43.0.10 -# +# services: etcd: backup_config: @@ -665,56 +476,63 @@ rancher_kubernetes_engine_config: service_node_port_range: 30000-32767 kube_controller: extra_args: - address: 127.0.0.1 feature-gates: RotateKubeletServerCertificate=true - profiling: 'false' - terminated-pod-gc-threshold: '1000' kubelet: extra_args: - anonymous-auth: 'false' - event-qps: '0' feature-gates: RotateKubeletServerCertificate=true - make-iptables-util-chains: 'true' protect-kernel-defaults: 'true' - streaming-connection-idle-timeout: 1800s tls-cipher-suites: >- TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 fail_swap_on: false generate_serving_certificate: true - scheduler: - extra_args: - address: 127.0.0.1 - profiling: 'false' ssh_agent_auth: false + upgrade_strategy: + max_unavailable_controlplane: '1' + max_unavailable_worker: 10% windows_prefered_cluster: false ``` -### Hardened Reference Ubuntu 18.04 LTS **cloud-config**: +### Hardened Reference Ubuntu 20.04 LTS **cloud-config**: The reference **cloud-config** is generally used in cloud infrastructure environments to allow for configuration management of compute instances. The reference config configures Ubuntu operating system level settings needed before installing kubernetes. -``` yaml +```yaml #cloud-config -packages: - - curl - - jq -runcmd: - - sysctl -w vm.overcommit_memory=1 - - sysctl -w kernel.panic=10 - - sysctl -w kernel.panic_on_oops=1 - - curl https://releases.rancher.com/install-docker/18.09.sh | sh - - usermod -aG docker ubuntu - - return=1; while [ $return != 0 ]; do sleep 2; docker ps; return=$?; done - - addgroup --gid 52034 etcd - - useradd --comment "etcd service account" --uid 52034 --gid 52034 etcd +apt: + sources: + docker.list: + source: deb [arch=amd64] http://download.docker.com/linux/ubuntu $RELEASE stable + keyid: 9DC858229FC7DD38854AE2D88D81803C0EBFCD88 +system_info: + default_user: + groups: + - docker write_files: - - path: /etc/sysctl.d/kubelet.conf - owner: root:root - permissions: "0644" - content: | - vm.overcommit_memory=1 - kernel.panic=10 - kernel.panic_on_oops=1 +- path: "/etc/apt/preferences.d/docker" + owner: root:root + permissions: '0600' + content: | + Package: docker-ce + Pin: version 5:19* + Pin-Priority: 800 +- path: "/etc/sysctl.d/90-kubelet.conf" + owner: root:root + permissions: '0644' + content: | + vm.overcommit_memory=1 + vm.panic_on_oom=0 + kernel.panic=10 + kernel.panic_on_oops=1 + kernel.keys.root_maxbytes=25000000 +package_update: true +packages: +- docker-ce +- docker-ce-cli +- containerd.io +runcmd: +- sysctl -p /etc/sysctl.d/90-kubelet.conf +- groupadd --gid 52034 etcd +- useradd --comment "etcd service account" --uid 52034 --gid 52034 etcd ```