diff --git a/.github/stale.yml b/.github/stale.yml new file mode 100644 index 00000000000..972e235a185 --- /dev/null +++ b/.github/stale.yml @@ -0,0 +1,39 @@ +# Configuration for probot-stale - https://github.com/probot/stale + +# Number of days of inactivity before an Issue or Pull Request becomes stale +daysUntilStale: 90 + +# Number of days of inactivity before an Issue or Pull Request with the stale label is closed. +# Set to false to disable. If disabled, issues still need to be closed manually, but will remain marked as stale. +daysUntilClose: 30 + +# Only issues or pull requests with all of these labels are check if stale. Defaults to `[]` (disabled) +onlyLabels: [] + +# Issues or Pull Requests with these labels will never be considered stale. Set to `[]` to disable +exemptLabels: [] + +# Set to true to ignore issues in a project (defaults to false) +exemptProjects: true + +# Set to true to ignore issues in a milestone (defaults to false) +exemptMilestones: true + +# Set to true to ignore issues with an assignee (defaults to false) +exemptAssignees: true + +# Label to use when marking as stale +staleLabel: status/stale + +# Comment to post when marking as stale. Set to `false` to disable +markComment: > + This repository uses a bot to automatically label issues which have not had any activity (commit/comment/label) + for 90 days. This helps us manage the community issues better. If the issue is still relevant, please add a comment to the + issue so the bot can remove the label and we know it is still valid. If it is no longer relevant (or possibly fixed in the + latest release), the bot will automatically close the issue in 30 days. Thank you for your contributions. + +# Limit the number of actions per hour, from 1-30. Default is 30 +limitPerRun: 30 + +# Limit to only `issues` +only: issues diff --git a/.gitignore b/.gitignore index daefacf85e2..e50d424efcf 100644 --- a/.gitignore +++ b/.gitignore @@ -11,3 +11,5 @@ package-lock.json /scripts/converters/results_to_markdown/.terraform /scripts/converters/results_to_markdown/terraform.tfstate* /scripts/converters/results_to_markdown/*.tfvars + +.idea/ diff --git a/Dockerfile.build b/Dockerfile.build index 614b1980fc8..a50a0e1e472 100644 --- a/Dockerfile.build +++ b/Dockerfile.build @@ -1,4 +1,4 @@ -FROM node:8-alpine +FROM node:16-alpine RUN apk update && apk add py-pygments bash git asciidoc gcompat && rm -rf /var/cache/apk/* diff --git a/config.toml b/config.toml index 35fdfda6e03..6ea4f5e3a0b 100644 --- a/config.toml +++ b/config.toml @@ -209,4 +209,4 @@ pre = "keyboard_arrow_down" [[menu.main]] name = "Partners" url = "https://rancher.com/partners/" - parent = "about" \ No newline at end of file + parent = "about" diff --git a/content/k3s/latest/en/_index.md b/content/k3s/latest/en/_index.md index a02147dc599..4d71be76b38 100644 --- a/content/k3s/latest/en/_index.md +++ b/content/k3s/latest/en/_index.md @@ -26,7 +26,7 @@ K3s is a fully compliant Kubernetes distribution with the following enhancements * Secure by default with reasonable defaults for lightweight environments. * Simple but powerful "batteries-included" features have been added, such as: a local storage provider, a service load balancer, a Helm controller, and the Traefik ingress controller. * Operation of all Kubernetes control plane components is encapsulated in a single binary and process. This allows K3s to automate and manage complex cluster operations like distributing certificates. -* External dependencies have been minimized (just a modern kernel and cgroup mounts needed). K3s packages required dependencies, including: +* External dependencies have been minimized (just a modern kernel and cgroup mounts needed). K3s packages the required dependencies, including: * containerd * Flannel * CoreDNS @@ -38,4 +38,4 @@ K3s is a fully compliant Kubernetes distribution with the following enhancements # What's with the name? -We wanted an installation of Kubernetes that was half the size in terms of memory footprint. Kubernetes is a 10-letter word stylized as K8s. So something half as big as Kubernetes would be a 5-letter word stylized as K3s. There is no long form of K3s and no official pronunciation. \ No newline at end of file +We wanted an installation of Kubernetes that was half the size in terms of memory footprint. Kubernetes is a 10-letter word stylized as K8s. So something half as big as Kubernetes would be a 5-letter word stylized as K3s. There is no long form of K3s and no official pronunciation. diff --git a/content/k3s/latest/en/advanced/_index.md b/content/k3s/latest/en/advanced/_index.md index 494650ebf82..554a0e5469a 100644 --- a/content/k3s/latest/en/advanced/_index.md +++ b/content/k3s/latest/en/advanced/_index.md @@ -13,18 +13,18 @@ This section contains advanced information describing the different ways you can - [Using Docker as the container runtime](#using-docker-as-the-container-runtime) - [Using etcdctl](#using-etcdctl) - [Configuring containerd](#configuring-containerd) -- [Secrets Encryption Config (Experimental)](#secrets-encryption-config-experimental) - [Running K3s with Rootless mode (Experimental)](#running-k3s-with-rootless-mode-experimental) - [Node labels and taints](#node-labels-and-taints) - [Starting the server with the installation script](#starting-the-server-with-the-installation-script) - [Additional preparation for Alpine Linux setup](#additional-preparation-for-alpine-linux-setup) +- [Additional preparation for (Red Hat/CentOS) Enterprise Linux](#additional-preparation-for-red-hat/centos-enterprise-linux) +- [Additional preparation for Raspberry Pi OS Setup](#additional-preparation-for-raspberry-pi-os-setup) +- [Enabling vxlan on Ubuntu 21.10+ on Raspberry Pi](#enabling-vxlan-on-ubuntu-21.10+-on-raspberry-pi) - [Running K3d (K3s in Docker) and docker-compose](#running-k3d-k3s-in-docker-and-docker-compose) -- [Enabling legacy iptables on Raspbian Buster](#enabling-legacy-iptables-on-raspbian-buster) -- [Enabling cgroups for Raspbian Buster](#enabling-cgroups-for-raspbian-buster) - [SELinux Support](#selinux-support) -- [Additional preparation for (Red Hat/CentOS) Enterprise Linux](#additional-preparation-for-red-hat-centos-enterprise-linux) - [Enabling Lazy Pulling of eStargz (Experimental)](#enabling-lazy-pulling-of-estargz-experimental) - [Additional Logging Sources](#additional-logging-sources) +- [Server and agent tokens](#server-and-agent-tokens) # Certificate Rotation @@ -127,8 +127,8 @@ If you would like to use etcdctl after installing K3s with embedded etcd, instal ``` $ VERSION="v3.5.0" -$ curl -L https://github.com/etcd-io/etcd/releases/download/${VERSION}/etcd-${VERSION}-linux-amd64.tar.gz --output etcdctl-${VERSION}-linux-amd64.tar.gz -$ sudo tar -zxvf etcdctl-${VERSION}-linux-amd64.tar.gz -C /usr/local/bin +$ curl -L https://github.com/etcd-io/etcd/releases/download/${VERSION}/etcd-${VERSION}-linux-amd64.tar.gz --output etcdctl-linux-amd64.tar.gz +$ sudo tar -zxvf etcdctl-linux-amd64.tar.gz --strip-components=1 -C /usr/local/bin etcd-${VERSION}-linux-amd64/etcdctl ``` Then start using etcdctl commands with the appropriate K3s flags: @@ -143,46 +143,8 @@ K3s will generate config.toml for containerd in `/var/lib/rancher/k3s/agent/etc/ For advanced customization for this file you can create another file called `config.toml.tmpl` in the same directory and it will be used instead. -The `config.toml.tmpl` will be treated as a Go template file, and the `config.Node` structure is being passed to the template. [This template](https://github.com/rancher/k3s/blob/master/pkg/agent/templates/templates.go#L16-L32) example on how to use the structure to customize the configuration file. +The `config.toml.tmpl` will be treated as a Go template file, and the `config.Node` structure is being passed to the template. See [this folder](https://github.com/k3s-io/k3s/blob/master/pkg/agent/templates) for Linux and Windows examples on how to use the structure to customize the configuration file. -# Secrets Encryption Config (Experimental) -As of v1.17.4+k3s1, K3s added the experimental feature of enabling secrets encryption at rest by passing the flag `--secrets-encryption` on a server, this flag will do the following automatically: - -- Generate an AES-CBC key -- Generate an encryption config file with the generated key - -``` -{ - "kind": "EncryptionConfiguration", - "apiVersion": "apiserver.config.k8s.io/v1", - "resources": [ - { - "resources": [ - "secrets" - ], - "providers": [ - { - "aescbc": { - "keys": [ - { - "name": "aescbckey", - "secret": "xxxxxxxxxxxxxxxxxxx" - } - ] - } - }, - { - "identity": {} - } - ] - } - ] -} -``` - -- Pass the config to the KubeAPI as encryption-provider-config - -Once enabled any created secret will be encrypted with this key. Note that if you disable encryption then any encrypted secrets will not be readable until you enable encryption again. # Running K3s with Rootless mode (Experimental) @@ -294,6 +256,39 @@ Then update the config and reboot: update-extlinux reboot ``` +# Additional preparation for (Red Hat/CentOS) Enterprise Linux + +It is recommended to turn off firewalld: +``` +systemctl disable firewalld --now +``` + +If enabled, it is required to disable nm-cloud-setup and reboot the node: +``` +systemctl disable nm-cloud-setup.service nm-cloud-setup.timer +reboot +``` + +# Additional preparation for Raspberry Pi OS Setup +## Enabling legacy iptables on Raspberry Pi OS +Raspberry Pi OS (formerly Raspbian) defaults to using `nftables` instead of `iptables`. **K3S** networking features require `iptables` and do not work with `nftables`. Follow the steps below to switch configure **Buster** to use `legacy iptables`: +``` +sudo iptables -F +sudo update-alternatives --set iptables /usr/sbin/iptables-legacy +sudo update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy +sudo reboot +``` + +## Enabling cgroups for Raspberry Pi OS + +Standard Raspberry Pi OS installations do not start with `cgroups` enabled. **K3S** needs `cgroups` to start the systemd service. `cgroups`can be enabled by appending `cgroup_memory=1 cgroup_enable=memory` to `/boot/cmdline.txt`. + +# Enabling vxlan on Ubuntu 21.10+ on Raspberry Pi + +Starting with Ubuntu 21.10, vxlan support on Raspberry Pi has been moved into a seperate kernel module. +``` +sudo apt install linux-modules-extra-raspi +``` # Running K3d (K3s in Docker) and docker-compose @@ -331,20 +326,6 @@ Alternatively the `docker run` command can also be used: --privileged rancher/k3s:vX.Y.Z -# Enabling legacy iptables on Raspbian Buster - -Raspbian Buster defaults to using `nftables` instead of `iptables`. **K3S** networking features require `iptables` and do not work with `nftables`. Follow the steps below to switch configure **Buster** to use `legacy iptables`: -``` -sudo iptables -F -sudo update-alternatives --set iptables /usr/sbin/iptables-legacy -sudo update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy -sudo reboot -``` - -# Enabling cgroups for Raspbian Buster - -Standard Raspbian Buster installations do not start with `cgroups` enabled. **K3S** needs `cgroups` to start the systemd service. `cgroups`can be enabled by appending `cgroup_memory=1 cgroup_enable=memory` to `/boot/cmdline.txt`. - ### example of /boot/cmdline.txt ``` console=serial0,115200 console=tty1 root=PARTUUID=58b06195-02 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait cgroup_memory=1 cgroup_enable=memory @@ -403,19 +384,6 @@ Using a custom `--data-dir` under SELinux is not supported. To customize it, you {{%/tab%}} {{% /tabs %}} -# Additional preparation for (Red Hat/CentOS) Enterprise Linux - -It is recommended to turn off firewalld: -``` -systemctl disable firewalld --now -``` - -If enabled, it is required to disable nm-cloud-setup and reboot the node: -``` -systemctl disable nm-cloud-setup.service nm-cloud-setup.timer -reboot -``` - # Enabling Lazy Pulling of eStargz (Experimental) ### What's lazy pulling and eStargz? @@ -479,4 +447,14 @@ helm repo add rancher-charts https://charts.rancher.io helm repo update helm install --create-namespace -n cattle-logging-system rancher-logging-crd rancher-charts/rancher-logging-crd helm install --create-namespace -n cattle-logging-system rancher-logging --set additionalLoggingSources.k3s.enabled=true rancher-charts/rancher-logging -``` \ No newline at end of file +``` + +# Server and agent tokens + +In K3s, there are two types of tokens: K3S_TOKEN and K3S_AGENT_TOKEN. + +K3S_TOKEN: Defines the key required by the server to offer the HTTP config resources. These resources are requested by the other servers before joining the K3s HA cluster. If the K3S_AGENT_TOKEN is not defined, the agents use this token as well to access the required HTTP resources to join the cluster. Note that this token is also used to generate the encryption key for important content in the database (e.g., bootstrap data). + +K3S_AGENT_TOKEN: Optional. Defines the key required by the server to offer HTTP config resources to the agents. If not defined, agents will require K3S_TOKEN. Defining K3S_AGENT_TOKEN is encouraged to avoid agents having to know K3S_TOKEN, which is also used to encrypt data. + +If no K3S_TOKEN is defined, the first K3s server will generate a random one. The result is part of the content in `/var/lib/rancher/k3s/server/token`. For example, `K1070878408e06a827960208f84ed18b65fa10f27864e71a57d9e053c4caff8504b::server:df54383b5659b9280aa1e73e60ef78fc`, where `df54383b5659b9280aa1e73e60ef78fc` is the K3S_TOKEN. diff --git a/content/k3s/latest/en/backup-restore/_index.md b/content/k3s/latest/en/backup-restore/_index.md index 27252dbb6fb..13031a06c35 100644 --- a/content/k3s/latest/en/backup-restore/_index.md +++ b/content/k3s/latest/en/backup-restore/_index.md @@ -18,7 +18,7 @@ For details on taking database snapshots and restoring your database from them, - [Official MySQL documentation](https://dev.mysql.com/doc/refman/8.0/en/replication-snapshot-method.html) - [Official PostgreSQL documentation](https://www.postgresql.org/docs/8.3/backup-dump.html) -- [Official etcd documentation](https://github.com/etcd-io/etcd/blob/master/Documentation/op-guide/recovery.md) +- [Official etcd documentation](https://etcd.io/docs/latest/op-guide/recovery/) # Backup and Restore with Embedded etcd Datastore (Experimental) @@ -57,7 +57,7 @@ These options can be passed in with the command line, or in the [configuration f | Options | Description | | ----------- | --------------- | | `--etcd-disable-snapshots` | Disable automatic etcd snapshots | -| `--etcd-snapshot-schedule-cron` value | Snapshot interval time in cron spec. eg. every 5 hours `* */5 * * *`(default: `0 */12 * * *`) | +| `--etcd-snapshot-schedule-cron` value | Snapshot interval time in cron spec. eg. every 5 hours `0 */5 * * *`(default: `0 */12 * * *`) | | `--etcd-snapshot-retention` value | Number of snapshots to retain (default: 5) | | `--etcd-snapshot-dir` value | Directory to save db snapshots. (Default location: `${data-dir}/db/snapshots`) | | `--cluster-reset` | Forget all peers and become sole member of a new cluster. This can also be set with the environment variable `[$K3S_CLUSTER_RESET]`. diff --git a/content/k3s/latest/en/helm/_index.md b/content/k3s/latest/en/helm/_index.md index 89d21d70ba5..5d43acbb78f 100644 --- a/content/k3s/latest/en/helm/_index.md +++ b/content/k3s/latest/en/helm/_index.md @@ -96,8 +96,9 @@ metadata: namespace: kube-system spec: valuesContent: |- - image: traefik - imageTag: v1.7.26-alpine + image: + name: traefik + tag: v2.6.1 proxyProtocol: enabled: true trustedIPs: diff --git a/content/k3s/latest/en/installation/airgap/_index.md b/content/k3s/latest/en/installation/airgap/_index.md index 91d37c00830..303cebab4b9 100644 --- a/content/k3s/latest/en/installation/airgap/_index.md +++ b/content/k3s/latest/en/installation/airgap/_index.md @@ -3,7 +3,7 @@ title: "Air-Gap Install" weight: 60 --- -You can install K3s in an air-gapped environment using two different methods. You can either deploy a private registry and mirror docker.io or you can manually deploy images such as for small clusters. +You can install K3s in an air-gapped environment using two different methods. An air-gapped environment is any environment that is not directly connected to the Internet. You can either deploy a private registry and mirror docker.io, or you can manually deploy images such as for small clusters. # Private Registry Method @@ -39,22 +39,23 @@ Follow the steps in the next section to install K3s. # Install K3s -Only after you have completed either the [Private Registry Method](#private-registry-method) or the [Manually Deploy Images Method](#manually-deploy-images-method) above should you install K3s. +### Prerequisites -Obtain the K3s binary from the [releases](https://github.com/rancher/k3s/releases) page, matching the same version used to get the airgap images. -Obtain the K3s install script at https://get.k3s.io +- Before installing K3s, complete the the [Private Registry Method](#private-registry-method) or the [Manually Deploy Images Method](#manually-deploy-images-method) above to prepopulate the images that K3s needs to install. +- Download the K3s binary from the [releases](https://github.com/rancher/k3s/releases) page, matching the same version used to get the airgap images. Place the binary in `/usr/local/bin` on each air-gapped node and ensure it is executable. +- Download the K3s install script at https://get.k3s.io. Place the install script anywhere on each air-gapped node, and name it `install.sh`. -Place the binary in `/usr/local/bin` on each node and ensure it is executable. -Place the install script anywhere on each node, and name it `install.sh`. +When running the K3s script with the `INSTALL_K3S_SKIP_DOWNLOAD` environment variable, K3s will use the local version of the script and binary. -### Install Options +### Installing K3s in an Air-Gapped Environment + You can install K3s on one or more servers as described below. {{% tabs %}} {{% tab "Single Server Configuration" %}} -To install K3s on a single server simply do the following on the server node. +To install K3s on a single server, simply do the following on the server node: ``` INSTALL_K3S_SKIP_DOWNLOAD=true ./install.sh @@ -75,13 +76,13 @@ For example, step two of the High Availability with an External DB guide mention ``` curl -sfL https://get.k3s.io | sh -s - server \ - --datastore-endpoint="mysql://username:password@tcp(hostname:3306)/database-name" + --datastore-endpoint='mysql://username:password@tcp(hostname:3306)/database-name' ``` Instead, you would modify such examples like below: ``` -INSTALL_K3S_SKIP_DOWNLOAD=true INSTALL_K3S_EXEC='server --datastore-endpoint="mysql://username:password@tcp(hostname:3306)/database-name"' ./install.sh +INSTALL_K3S_SKIP_DOWNLOAD=true INSTALL_K3S_EXEC='server' K3S_DATASTORE_ENDPOINT='mysql://username:password@tcp(hostname:3306)/database-name' ./install.sh ``` {{% /tab %}} diff --git a/content/k3s/latest/en/installation/datastore/_index.md b/content/k3s/latest/en/installation/datastore/_index.md index 059d73e16fe..b5476b42f35 100644 --- a/content/k3s/latest/en/installation/datastore/_index.md +++ b/content/k3s/latest/en/installation/datastore/_index.md @@ -23,7 +23,7 @@ If you wish to use an external datastore such as PostgreSQL, MySQL, or etcd you CLI Flag | Environment Variable | Description ------------|-------------|------------------ - `--datastore-endpoint` | `K3S_DATASTORE_ENDPOINT` | Specify a PostgresSQL, MySQL, or etcd connection string. This is a string used to describe the connection to the datastore. The structure of this string is specific to each backend and is detailed below. + `--datastore-endpoint` | `K3S_DATASTORE_ENDPOINT` | Specify a PostgreSQL, MySQL, or etcd connection string. This is a string used to describe the connection to the datastore. The structure of this string is specific to each backend and is detailed below. `--datastore-cafile` | `K3S_DATASTORE_CAFILE` | TLS Certificate Authority (CA) file used to help secure communication with the datastore. If your datastore serves requests over TLS using a certificate signed by a custom certificate authority, you can specify that CA using this parameter so that the K3s client can properly verify the certificate. | | `--datastore-certfile` | `K3S_DATASTORE_CERTFILE` | TLS certificate file used for client certificate based authentication to your datastore. To use this feature, your datastore must be configured to support client certificate based authentication. If you specify this parameter, you must also specify the `datastore-keyfile` parameter. | | `--datastore-keyfile` | `K3S_DATASTORE_KEYFILE` | TLS key file used for client certificate based authentication to your datastore. See the previous `datastore-certfile` parameter for more details. | @@ -81,7 +81,7 @@ The above assumes a typical three node etcd cluster. The parameter can accept on {{% /tab %}} {{% /tabs %}} -
Based on the above, the following example command could be used to launch a server instance that connects to a PostgresSQL database named k3s: +
Based on the above, the following example command could be used to launch a server instance that connects to a PostgreSQL database named k3s: ``` K3S_DATASTORE_ENDPOINT='postgres://username:password@hostname:5432/k3s' k3s server ``` diff --git a/content/k3s/latest/en/installation/disable-flags/_index.md b/content/k3s/latest/en/installation/disable-flags/_index.md index 83b6b692cca..2dbbfeb8920 100644 --- a/content/k3s/latest/en/installation/disable-flags/_index.md +++ b/content/k3s/latest/en/installation/disable-flags/_index.md @@ -3,7 +3,7 @@ title: "Disable Components Flags" weight: 60 --- -When starting K3s server with --cluster-init it will run all control plane components that includes (api server, controller manager, scheduler, and etcd). However you can run server nodes with certain components and execlude others, the following sections will explain how to do that. +Starting the K3s server with `--cluster-init` will run all control plane components, including the api server, controller manager, scheduler, and etcd. However, you can run server nodes with certain components and exclude others; the following sections will explain how to do that. # ETCD Only Nodes diff --git a/content/k3s/latest/en/installation/ha-embedded/_index.md b/content/k3s/latest/en/installation/ha-embedded/_index.md index f8fe7d2ebea..6d4bb7bab49 100644 --- a/content/k3s/latest/en/installation/ha-embedded/_index.md +++ b/content/k3s/latest/en/installation/ha-embedded/_index.md @@ -23,3 +23,14 @@ K3S_TOKEN=SECRET k3s server --server https://:6443 ``` Now you have a highly available control plane. Joining additional worker nodes to the cluster follows the same procedure as a single server cluster. + +There are a few config flags that must be the same in all server nodes: + +* Network related flags: `--cluster-dns`, `--cluster-domain`, `--cluster-cidr`, `--service-cidr` +* Flags controlling the deployment of certain components: `--disable-helm-controller`, `--disable-kube-proxy`, `--disable-network-policy` and any component passed to `--disable` +* Feature related flags: `--secrets-encryption` + +## Existing clusters +If you have an existing cluster using the default embedded SQLite database, you can convert it to etcd by simply restarting your K3s server with the `--cluster-init` flag. Once you've done that, you'll be able to add additional instances as described above. + +>**Important:** K3s v1.22.2 and newer support migration from SQLite to etcd. Older versions will create a new empty datastore if you add `--cluster-init` to an existing server. diff --git a/content/k3s/latest/en/installation/ha/_index.md b/content/k3s/latest/en/installation/ha/_index.md index 00c082d2ac5..4e5072bfff8 100644 --- a/content/k3s/latest/en/installation/ha/_index.md +++ b/content/k3s/latest/en/installation/ha/_index.md @@ -33,12 +33,13 @@ You will first need to create an external datastore for the cluster. See the [Cl ### 2. Launch Server Nodes K3s requires two or more server nodes for this HA configuration. See the [Installation Requirements]({{}}/k3s/latest/en/installation/installation-requirements/) guide for minimum machine requirements. -When running the `k3s server` command on these nodes, you must set the `datastore-endpoint` parameter so that K3s knows how to connect to the external datastore. +When running the `k3s server` command on these nodes, you must set the `datastore-endpoint` parameter so that K3s knows how to connect to the external datastore. The `token` parameter can also be used to set a deterministic token when adding nodes. When empty, this token will be generated automatically for further use. -For example, a command like the following could be used to install the K3s server with a MySQL database as the external datastore: +For example, a command like the following could be used to install the K3s server with a MySQL database as the external datastore and [set a token]({{}}/k3s/latest/en/installation/install-options/server-config/#cluster-options): -``` +```bash curl -sfL https://get.k3s.io | sh -s - server \ + --token=SECRET \ --datastore-endpoint="mysql://username:password@tcp(hostname:3306)/database-name" ``` @@ -62,12 +63,37 @@ Agent nodes need a URL to register against. This can be the IP or hostname of an This endpoint can also be used for accessing the Kubernetes API. So you can, for example, modify your [kubeconfig](https://kubernetes.io/docs/concepts/configuration/organize-cluster-access-kubeconfig/) file to point to it instead of a specific node. To avoid certificate errors in such a configuration, you should install the server with the `--tls-san YOUR_IP_OR_HOSTNAME_HERE` option. This option adds an additional hostname or IP as a Subject Alternative Name in the TLS cert, and it can be specified multiple times if you would like to access via both the IP and the hostname. -### 4. Optional: Join Agent Nodes +### 4. Optional: Join Additional Server Nodes + +The same example command in Step 2 can be used to join additional server nodes, where the token from the first node needs to be used. + +If the first server node was started without the `--token` CLI flag or `K3S_TOKEN` variable, the token value can be retrieved from any server already joined to the cluster: +```bash +cat /var/lib/rancher/k3s/server/token +``` + +Additional server nodes can then be added [using the token]({{}}/k3s/latest/en/installation/install-options/server-config/#cluster-options): + +```bash +curl -sfL https://get.k3s.io | sh -s - server \ + --token=SECRET \ + --datastore-endpoint="mysql://username:password@tcp(hostname:3306)/database-name" +``` + +There are a few config flags that must be the same in all server nodes: + +* Network related flags: `--cluster-dns`, `--cluster-domain`, `--cluster-cidr`, `--service-cidr` +* Flags controlling the deployment of certain components: `--disable-helm-controller`, `--disable-kube-proxy`, `--disable-network-policy` and any component passed to `--disable` +* Feature related flags: `--secrets-encryption` + +> **Note:** Ensure that you retain a copy of this token as it is required when restoring from backup and adding nodes. Previously, K3s did not enforce the use of a token when using external SQL datastores. + +### 5. Optional: Join Agent Nodes Because K3s server nodes are schedulable by default, the minimum number of nodes for an HA K3s server cluster is two server nodes and zero agent nodes. To add nodes designated to run your apps and services, join agent nodes to your cluster. Joining agent nodes in an HA cluster is the same as joining agent nodes in a single server cluster. You just need to specify the URL the agent should register to and the token it should use. -``` +```bash K3S_TOKEN=SECRET k3s agent --server https://fixed-registration-address:6443 ``` diff --git a/content/k3s/latest/en/installation/install-options/how-to-flags/_index.md b/content/k3s/latest/en/installation/install-options/how-to-flags/_index.md index 25aa9b43567..447dfdc40f9 100644 --- a/content/k3s/latest/en/installation/install-options/how-to-flags/_index.md +++ b/content/k3s/latest/en/installation/install-options/how-to-flags/_index.md @@ -27,9 +27,23 @@ If this command is not specified as a server or agent command, it will default t The final systemd command resolves to a combination of this environment variable and script args. To illustrate this, the following commands result in the same behavior of registering a server without flannel: ```bash -curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--no-flannel" sh -s - -curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="server --no-flannel" sh -s - -curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="server" sh -s - --no-flannel -curl -sfL https://get.k3s.io | sh -s - server --no-flannel -curl -sfL https://get.k3s.io | sh -s - --no-flannel -``` \ No newline at end of file +curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--flannel-backend none" sh -s - +curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="server --flannel-backend none" sh -s - +curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="server" sh -s - --flannel-backend none +curl -sfL https://get.k3s.io | sh -s - server --flannel-backend none +curl -sfL https://get.k3s.io | sh -s - --flannel-backend none +``` + +### Example C: CONFIG FILE + +Before installing k3s, you can create a file called `config.yaml` containing fields that match CLI flags. That file needs to be in the path: `/etc/rancher/k3s/config.yaml` for k3s to consume it. + +The fields in the config file drop the starting `--` from the matching CLI flag. For example: + +``` +write-kubeconfig-mode: 644 +token: "secret" +node-ip: 10.0.10.22,2a05:d012:c6f:4655:d73c:c825:a184:1b75 +cluster-cidr: 10.42.0.0/16,2001:cafe:42:0::/56 +service-cidr: 10.43.0.0/16,2001:cafe:42:1::/112 +``` diff --git a/content/k3s/latest/en/installation/install-options/server-config/_index.md b/content/k3s/latest/en/installation/install-options/server-config/_index.md index 687133583d8..853d2ccb491 100644 --- a/content/k3s/latest/en/installation/install-options/server-config/_index.md +++ b/content/k3s/latest/en/installation/install-options/server-config/_index.md @@ -152,6 +152,7 @@ the agent options are there because the server has the agent process embedded wi | Flag | Description | |------|--------------| +| `--etcd-arg` value | Customized flag for etcd process | | `--kube-apiserver-arg` value | Customized flag for kube-apiserver process | | `--kube-scheduler-arg` value | Customized flag for kube-scheduler process | | `--kube-controller-manager-arg` value | Customized flag for kube-controller-manager process | @@ -234,6 +235,7 @@ OPTIONS: --token-file value (cluster) File containing the cluster-secret/token [$K3S_TOKEN_FILE] --write-kubeconfig value, -o value (client) Write kubeconfig for admin client to this file [$K3S_KUBECONFIG_OUTPUT] --write-kubeconfig-mode value (client) Write kubeconfig with this mode [$K3S_KUBECONFIG_MODE] + --etcd-arg value (flags) Customized flag for etcd process --kube-apiserver-arg value (flags) Customized flag for kube-apiserver process --kube-scheduler-arg value (flags) Customized flag for kube-scheduler process --kube-controller-manager-arg value (flags) Customized flag for kube-controller-manager process diff --git a/content/k3s/latest/en/installation/installation-requirements/_index.md b/content/k3s/latest/en/installation/installation-requirements/_index.md index 1b5d14825de..f4fd336d7f7 100644 --- a/content/k3s/latest/en/installation/installation-requirements/_index.md +++ b/content/k3s/latest/en/installation/installation-requirements/_index.md @@ -44,13 +44,14 @@ K3s performance depends on the performance of the database. To ensure optimal sp The K3s server needs port 6443 to be accessible by all nodes. -The nodes need to be able to reach other nodes over UDP port 8472 when Flannel VXLAN is used. The node should not listen on any other port. K3s uses reverse tunneling such that the nodes make outbound connections to the server and all kubelet traffic runs through that tunnel. However, if you do not use Flannel and provide your own custom CNI, then port 8472 is not needed by K3s. +The nodes need to be able to reach other nodes over UDP port 8472 when Flannel VXLAN is used or over UDP ports 51820 and 51821 (when using IPv6) when Flannel Wireguard backend is used. The node should not listen on any other port. K3s uses reverse tunneling such that the nodes make outbound connections to the server and all kubelet traffic runs through that tunnel. However, if you do not use Flannel and provide your own custom CNI, then the ports needed by Flannel are not needed by K3s. If you wish to utilize the metrics server, you will need to open port 10250 on each node. If you plan on achieving high availability with embedded etcd, server nodes must be accessible to each other on ports 2379 and 2380. > **Important:** The VXLAN port on nodes should not be exposed to the world as it opens up your cluster network to be accessed by anyone. Run your nodes behind a firewall/security group that disables access to port 8472. +> **Warning:** Flannel relies on the [Bridge CNI plugin](https://www.cni.dev/plugins/current/main/bridge/) to create a L2 network that switches traffic. Rogue pods with NET_RAW capabilities can abuse that L2 network to launch attacks such as [ARP spoofing](https://static.sched.com/hosted_files/kccncna19/72/ARP%20DNS%20spoof.pdf). Therefore, as documented in the [kubernetes docs](https://kubernetes.io/docs/concepts/security/pod-security-standards/), please set a restricted profile that disables NET_RAW on non-trustable pods.
Inbound Rules for K3s Server Nodes
@@ -58,6 +59,8 @@ If you plan on achieving high availability with embedded etcd, server nodes must |-----|-----|----------------|---| | TCP | 6443 | K3s agent nodes | Kubernetes API Server | UDP | 8472 | K3s server and agent nodes | Required only for Flannel VXLAN +| UDP | 51820 | K3s server and agent nodes | Required only for Flannel Wireguard backend +| UDP | 51821 | K3s server and agent nodes | Required only for Flannel Wireguard backend with IPv6 | TCP | 10250 | K3s server and agent nodes | Kubelet metrics | TCP | 2379-2380 | K3s server nodes | Required only for HA with embedded etcd diff --git a/content/k3s/latest/en/installation/network-options/_index.md b/content/k3s/latest/en/installation/network-options/_index.md index dcc65a03aea..652f22f6adc 100644 --- a/content/k3s/latest/en/installation/network-options/_index.md +++ b/content/k3s/latest/en/installation/network-options/_index.md @@ -19,10 +19,11 @@ If you wish to use WireGuard as your flannel backend it may require additional k `--flannel-backend=ipsec` | Uses the IPSEC backend which encrypts network traffic. | `--flannel-backend=host-gw` | Uses the host-gw backend. | `--flannel-backend=wireguard` | Uses the WireGuard backend which encrypts network traffic. May require additional kernel modules and configuration. | + `--flannel-ipv6-masq` | Apply masquerading rules to IPv6 traffic (default for IPv4). Only applies on dual-stack or IPv6-only clusters | ### Custom CNI -Run K3s with `--flannel-backend=none` and install your CNI of choice. IP Forwarding should be enabled for Canal and Calico. Please reference the steps below. +Run K3s with `--flannel-backend=none` and install your CNI of choice. Most CNI plugins come with their own network policy engine, so it is recommended to set `--disable-network-policy` as well to avoid conflicts. IP Forwarding should be enabled for Canal and Calico. Please reference the steps below. {{% tabs %}} {{% tab "Canal" %}} @@ -74,15 +75,22 @@ You should see that IP forwarding is set to true. Dual-stack networking must be configured when the cluster is first created. It cannot be enabled on an existing single-stack cluster. -To enable dual-stack in k3s, you must provide valid dual-stack `cluster-cidr` and `service-cidr`, and set `disable-network-policy` on all server nodes. Both servers and agents must provide valid dual-stack `node-ip` settings. Node address auto-detection and network policy enforcement are not supported on dual-stack clusters when using the default flannel CNI. Besides, only vxlan backend is supported at the moment. This is an example of a valid configuration: +Dual-stack is supported on k3s v1.21 or above. + +To enable dual-stack in K3s, you must provide valid dual-stack `cluster-cidr` and `service-cidr` on all server nodes. Both servers and agents must provide valid dual-stack `node-ip` settings. Node address auto-detection is not supported on dual-stack clusters, because kubelet fetches only the first IP address that it finds. Additionally, only vxlan backend is supported currently. This is an example of a valid configuration: ``` -node-ip: 10.0.10.7,2a05:d012:c6f:4611:5c2:5602:eed2:898c -cluster-cidr: 10.42.0.0/16,2001:cafe:42:0::/56 -service-cidr: 10.43.0.0/16,2001:cafe:42:1::/112 -disable-network-policy: true +k3s server --node-ip 10.0.10.7,2a05:d012:c6f:4611:5c2:5602:eed2:898c --cluster-cidr 10.42.0.0/16,2001:cafe:42:0::/56 --service-cidr 10.43.0.0/16,2001:cafe:42:1::/112 ``` Note that you can choose whatever `cluster-cidr` and `service-cidr` value, however the `node-ip` values must correspond to the ip addresses of your main interface. Remember to allow ipv6 traffic if you are deploying in a public cloud. If you are using a custom cni plugin, i.e. a cni plugin different from flannel, the previous configuration might not be enough to enable dual-stack in the cni plugin. Please check how to enable dual-stack in its documentation and verify if network policies can be enabled. + +### IPv6 only installation + +IPv6 only setup is supported on k3s v1.22 or above. Note that network policy enforcement is not supported on IPv6-only clusters when using the default flannel CNI. This is an example of a valid configuration: + +``` +k3s server --disable-network-policy +``` diff --git a/content/k3s/latest/en/installation/private-registry/_index.md b/content/k3s/latest/en/installation/private-registry/_index.md index c87ee3df0cb..523ae6c8586 100644 --- a/content/k3s/latest/en/installation/private-registry/_index.md +++ b/content/k3s/latest/en/installation/private-registry/_index.md @@ -32,22 +32,43 @@ mirrors: Each mirror must have a name and set of endpoints. When pulling an image from a registry, containerd will try these endpoint URLs one by one, and use the first working one. +#### Rewrites + +Each mirror can have a set of rewrites. Rewrites can change the tag of an image based on a regular expression. This is useful if the organization/project structure in the mirror registry is different to the upstream one. + +For example, the following configuration would transparently pull the image `docker.io/rancher/coredns-coredns:1.6.3` from `registry.example.com:5000/mirrorproject/rancher-images/coredns-coredns:1.6.3`: + +``` +mirrors: + docker.io: + endpoint: + - "https://registry.example.com:5000" + rewrite: + "^rancher/(.*)": "mirrorproject/rancher-images/$1" +``` + +The image will still be stored under the original name so that a `crictl image ls` will show `docker.io/rancher/coredns-coredns:1.6.3` as available on the node, even though the image was pulled from the mirrored registry with a different name. + ### Configs -The configs section defines the TLS and credential configuration for each mirror. For each mirror you can define `auth` and/or `tls`. The TLS part consists of: +The `configs` section defines the TLS and credential configuration for each mirror. For each mirror you can define `auth` and/or `tls`. -Directive | Description -----------|------------ -`cert_file` | The client certificate path that will be used to authenticate with the registry -`key_file` | The client key path that will be used to authenticate with the registry -`ca_file` | Defines the CA certificate path to be used to verify the registry's server cert file -`insecure_skip_verify` | Boolean that defines if TLS verification should be skipped for the registry +The `tls` part consists of: -The credentials consist of either username/password or authentication token: +| Directive | Description | +|------------------------|--------------------------------------------------------------------------------------| +| `cert_file` | The client certificate path that will be used to authenticate with the registry | +| `key_file` | The client key path that will be used to authenticate with the registry | +| `ca_file` | Defines the CA certificate path to be used to verify the registry's server cert file | +| `insecure_skip_verify` | Boolean that defines if TLS verification should be skipped for the registry | -- username: user name of the private registry basic auth -- password: user password of the private registry basic auth -- auth: authentication token of the private registry basic auth +The `auth` part consists of either username/password or authentication token: + +| Directive | Description | +|------------|---------------------------------------------------------| +| `username` | user name of the private registry basic auth | +| `password` | user password of the private registry basic auth | +| `auth` | authentication token of the private registry basic auth | Below are basic examples of using private registries in different modes: @@ -142,4 +163,4 @@ Example: `docker tag coredns-coredns:1.6.3 mycustomreg:5000/coredns-coredns` Last, push the images to the private registry. -Example: `docker push mycustomreg:5000/coredns-coredns` +Example: `docker push mycustomreg.com:5000/coredns-coredns` diff --git a/content/k3s/latest/en/networking/_index.md b/content/k3s/latest/en/networking/_index.md index 49ed55deb46..2dff22e927d 100644 --- a/content/k3s/latest/en/networking/_index.md +++ b/content/k3s/latest/en/networking/_index.md @@ -42,7 +42,7 @@ To migrate from an older Traefik v1 instance please refer to the [Traefik docume # Service Load Balancer -Any service load balancer (LB) can be leveraged in your Kubernetes cluster. K3s provides a load balancer known as [Klipper Load Balancer](https://github.com/rancher/klipper-lb) that uses available host ports. +Any service load balancer (LB) can be leveraged in your Kubernetes cluster. K3s provides a load balancer known as [Klipper Load Balancer](https://github.com/k3s-io/klipper-lb) that uses available host ports. Upstream Kubernetes allows a Service of type LoadBalancer to be created, but doesn't include the implementation of the LB. Some LB services require a cloud provider such as Amazon EC2 or Microsoft Azure. By contrast, the K3s service LB makes it possible to use an LB service without a cloud provider. diff --git a/content/k3s/latest/en/security/_index.md b/content/k3s/latest/en/security/_index.md index f8b4285fc49..ba6ef7ccbd6 100644 --- a/content/k3s/latest/en/security/_index.md +++ b/content/k3s/latest/en/security/_index.md @@ -5,7 +5,7 @@ weight: 90 This section describes the methodology and means of securing a K3s cluster. It's broken into 2 sections. These guides assume k3s is running with embedded etcd. -The documents below apply to both CIS 1.5 & 1.6. +The documents below apply to CIS Kubernetes Benchmark v1.6. * [Hardening Guide](./hardening_guide/) * [CIS Benchmark Self-Assessment Guide](./self_assessment/) diff --git a/content/k3s/latest/en/security/hardening_guide/_index.md b/content/k3s/latest/en/security/hardening_guide/_index.md index 6a66acc02aa..e22571f30ef 100644 --- a/content/k3s/latest/en/security/hardening_guide/_index.md +++ b/content/k3s/latest/en/security/hardening_guide/_index.md @@ -3,12 +3,12 @@ title: "CIS Hardening Guide" weight: 80 --- -This document provides prescriptive guidance for hardening a production installation of K3s. It outlines the configurations and controls required to address Kubernetes benchmark controls from the Center for Information Security (CIS). +This document provides prescriptive guidance for hardening a production installation of K3s. It outlines the configurations and controls required to address Kubernetes benchmark controls from the Center for Internet Security (CIS). K3s has a number of security mitigations applied and turned on by default and will pass a number of the Kubernetes CIS controls without modification. There are some notable exceptions to this that require manual intervention to fully comply with the CIS Benchmark: 1. K3s will not modify the host operating system. Any host-level modifications will need to be done manually. -2. Certain CIS policy controls for PodSecurityPolicies and NetworkPolicies will restrict the functionality of this cluster. You must opt into having K3s configure these by adding the appropriate options (enabling of admission plugins) to your command-line flags or configuration file as well as manually applying appropriate policies. Further detail in the sections below. +2. Certain CIS policy controls for `PodSecurityPolicies` and `NetworkPolicies` will restrict the functionality of the cluster. You must opt into having K3s configure these by adding the appropriate options (enabling of admission plugins) to your command-line flags or configuration file as well as manually applying appropriate policies. Further details are presented in the sections below. The first section (1.1) of the CIS Benchmark concerns itself primarily with pod manifest permissions and ownership. K3s doesn't utilize these for the core components since everything is packaged into a single binary. @@ -31,23 +31,24 @@ vm.panic_on_oom=0 vm.overcommit_memory=1 kernel.panic=10 kernel.panic_on_oops=1 +kernel.keys.root_maxbytes=25000000 ``` ## Kubernetes Runtime Requirements -The runtime requirements to comply with the CIS Benchmark are centered around pod security (PSPs) and network policies. These are outlined in this section. K3s doesn't apply any default PSPs or network policies however K3s ships with a controller that is meant to apply a given set of network policies. By default, K3s runs with the "NodeRestriction" admission controller. To enable PSPs, add the following to the K3s start command: `--kube-apiserver-arg="enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount"`. This will have the effect of maintaining the "NodeRestriction" plugin as well as enabling the "PodSecurityPolicy". +The runtime requirements to comply with the CIS Benchmark are centered around pod security (PSPs), network policies and API Server auditing logs. These are outlined in this section. K3s doesn't apply any default PSPs or network policies. However, K3s ships with a controller that is meant to apply a given set of network policies. By default, K3s runs with the `NodeRestriction` admission controller. To enable PSPs, add the following to the K3s start command: `--kube-apiserver-arg="enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount"`. This will have the effect of maintaining the `NodeRestriction` plugin as well as enabling the `PodSecurityPolicy`. The same happens with the API Server auditing logs, K3s doesn't enable them by default, so audit log configuration and audit policy must be created manually. -### PodSecurityPolicies +### Pod Security Policies When PSPs are enabled, a policy can be applied to satisfy the necessary controls described in section 5.2 of the CIS Benchmark. -Here's an example of a compliant PSP. +Here is an example of a compliant PSP. ```yaml apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: - name: cis1.5-compliant-psp + name: restricted-psp spec: privileged: false # CIS - 5.2.1 allowPrivilegeEscalation: false # CIS - 5.2.5 @@ -59,7 +60,9 @@ spec: - 'projected' - 'secret' - 'downwardAPI' + - 'csi' - 'persistentVolumeClaim' + - 'ephemeral' hostNetwork: false # CIS - 5.2.4 hostIPC: false # CIS - 5.2.3 hostPID: false # CIS - 5.2.2 @@ -80,7 +83,7 @@ spec: readOnlyRootFilesystem: false ``` -Before the above PSP to be effective, we need to create a couple ClusterRoles and ClusterRole. We also need to include a "system unrestricted policy" which is needed for system-level pods that require additional privileges. +For the above PSP to be effective, we need to create a ClusterRole and a ClusterRoleBinding. We also need to include a "system unrestricted policy" which is needed for system-level pods that require additional privileges. These can be combined with the PSP yaml above and NetworkPolicy yaml below into a single file and placed in the `/var/lib/rancher/k3s/server/manifests` directory. Below is an example of a `policy.yaml` file. @@ -88,7 +91,7 @@ These can be combined with the PSP yaml above and NetworkPolicy yaml below into apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: - name: cis1.5-compliant-psp + name: restricted-psp spec: privileged: false allowPrivilegeEscalation: false @@ -100,7 +103,9 @@ spec: - 'projected' - 'secret' - 'downwardAPI' + - 'csi' - 'persistentVolumeClaim' + - 'ephemeral' hostNetwork: false hostIPC: false hostPID: false @@ -123,7 +128,7 @@ spec: apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: - name: psp:restricted + name: psp:restricted-psp labels: addonmanager.kubernetes.io/mode: EnsureExists rules: @@ -131,62 +136,23 @@ rules: resources: ['podsecuritypolicies'] verbs: ['use'] resourceNames: - - cis1.5-compliant-psp + - restricted-psp --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: - name: default:restricted + name: default:restricted-psp labels: addonmanager.kubernetes.io/mode: EnsureExists roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole - name: psp:restricted + name: psp:restricted-psp subjects: - kind: Group name: system:authenticated apiGroup: rbac.authorization.k8s.io --- -kind: NetworkPolicy -apiVersion: networking.k8s.io/v1 -metadata: - name: intra-namespace - namespace: kube-system -spec: - podSelector: {} - ingress: - - from: - - namespaceSelector: - matchLabels: - name: kube-system ---- -kind: NetworkPolicy -apiVersion: networking.k8s.io/v1 -metadata: - name: intra-namespace - namespace: default -spec: - podSelector: {} - ingress: - - from: - - namespaceSelector: - matchLabels: - name: default ---- -kind: NetworkPolicy -apiVersion: networking.k8s.io/v1 -metadata: - name: intra-namespace - namespace: kube-public -spec: - podSelector: {} - ingress: - - from: - - namespaceSelector: - matchLabels: - name: kube-public ---- apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: @@ -253,6 +219,45 @@ subjects: - apiGroup: rbac.authorization.k8s.io kind: Group name: system:serviceaccounts +--- +kind: NetworkPolicy +apiVersion: networking.k8s.io/v1 +metadata: + name: intra-namespace + namespace: kube-system +spec: + podSelector: {} + ingress: + - from: + - namespaceSelector: + matchLabels: + name: kube-system +--- +kind: NetworkPolicy +apiVersion: networking.k8s.io/v1 +metadata: + name: intra-namespace + namespace: default +spec: + podSelector: {} + ingress: + - from: + - namespaceSelector: + matchLabels: + name: default +--- +kind: NetworkPolicy +apiVersion: networking.k8s.io/v1 +metadata: + name: intra-namespace + namespace: kube-public +spec: + podSelector: {} + ingress: + - from: + - namespaceSelector: + matchLabels: + name: kube-public ``` > **Note:** The Kubernetes critical additions such as CNI, DNS, and Ingress are ran as pods in the `kube-system` namespace. Therefore, this namespace will have a policy that is less restrictive so that these components can run properly. @@ -263,7 +268,7 @@ subjects: CIS requires that all namespaces have a network policy applied that reasonably limits traffic into namespaces and pods. -Here's an example of a compliant network policy. +Here is an example of a compliant network policy. ```yaml kind: NetworkPolicy @@ -302,11 +307,25 @@ spec: - Ingress ``` -If you are using the default traefik ingress controller with k3s, it will also be blocked by default, so the following network policies must be added to allow traffic to both traefik pods and svclb pods in the kube-system namespace. For version 1.20 and below there is a different label `traefik` used than in 1.21 and above, so remove the one that is not associated with your Kubernetes version. +The metrics-server and Traefik ingress controller will be blocked by default if network policies are not created to allow access. Traefik v1 as packaged in K3s version 1.20 and below uses different labels than Traefik v2. Ensure that you only use the sample yaml below that is associated with the version of Traefik present on your cluster. ```yaml apiVersion: networking.k8s.io/v1 kind: NetworkPolicy +metadata: + name: allow-all-metrics-server + namespace: kube-system +spec: + podSelector: + matchLabels: + k8s-app: metrics-server + ingress: + - {} + policyTypes: + - Ingress +--- +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy metadata: name: allow-all-svclbtraefik-ingress namespace: kube-system @@ -319,7 +338,7 @@ spec: policyTypes: - Ingress --- -# 1.20 +# Below is for 1.20 ONLY -- remove if on 1.21 or above apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: @@ -334,7 +353,7 @@ spec: policyTypes: - Ingress --- -# 1.21 +# Below is for 1.21 and above ONLY -- remove if on 1.20 or below apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: @@ -352,24 +371,67 @@ spec: > **Note:** Operators must manage network policies as normal for additional namespaces that are created. +### API Server audit configuration + +CIS requirements 1.2.22 to 1.2.25 are related to configuring audit logs for the API Server. K3s doesn't create by default the log directory and audit policy, as auditing requirements are specific to each user's policies and environment. + +The log directory, ideally, must be created before starting K3s. A restrictive access permission is recommended to avoid leaking potential sensitive information. + +```bash +sudo mkdir -p -m 700 /var/lib/rancher/k3s/server/logs +``` + +A starter audit policy to log request metadata is provided below. The policy should be written to a file named `audit.yaml` in `/var/lib/rancher/k3s/server` directory. Detailed information about policy configuration for the API server can be found in the Kubernetes [documentation](https://kubernetes.io/docs/tasks/debug-application-cluster/audit/). + +```yaml +apiVersion: audit.k8s.io/v1 +kind: Policy +rules: +- level: Metadata +``` + +Both configurations must be passed as arguments to the API Server as: + +```bash +--kube-apiserver-arg='audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log' +--kube-apiserver-arg='audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml' +``` + +If the configurations are created after K3s is installed, they must be added to K3s' systemd service in `/etc/systemd/system/k3s.service`. + +```bash +ExecStart=/usr/local/bin/k3s \ + server \ + '--kube-apiserver-arg=audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log' \ + '--kube-apiserver-arg=audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml' \ +``` + +K3s must be restarted to load the new configuration. + +```bash +sudo systemctl daemon-reload +sudo systemctl restart k3s.service +``` + +Additional information about CIS requirements 1.2.22 to 1.2.25 is presented below. + ## Known Issues The following are controls that K3s currently does not pass by default. Each gap will be explained, along with a note clarifying whether it can be passed through manual operator intervention, or if it will be addressed in a future release of K3s. - ### Control 1.2.15 Ensure that the admission control plugin `NamespaceLifecycle` is set.
Rationale -Setting admission control policy to NamespaceLifecycle ensures that objects cannot be created in non-existent namespaces, and that namespaces undergoing termination are not used for creating the new objects. This is recommended to enforce the integrity of the namespace termination process and also for the availability of the newer objects. +Setting admission control policy to `NamespaceLifecycle` ensures that objects cannot be created in non-existent namespaces, and that namespaces undergoing termination are not used for creating the new objects. This is recommended to enforce the integrity of the namespace termination process and also for the availability of the newer objects. This can be remediated by passing this argument as a value to the `enable-admission-plugins=` and pass that to `--kube-apiserver-arg=` argument to `k3s server`. An example can be found below.
-### Control 1.2.16 (mentioned above) +### Control 1.2.16 Ensure that the admission control plugin `PodSecurityPolicy` is set.
Rationale -A Pod Security Policy is a cluster-level resource that controls the actions that a pod can perform and what it has the ability to access. The PodSecurityPolicy objects define a set of conditions that a pod must run with in order to be accepted into the system. Pod Security Policies are comprised of settings and strategies that control the security features a pod has access to and hence this must be used to control pod access permissions. +A Pod Security Policy is a cluster-level resource that controls the actions that a pod can perform and what it has the ability to access. The `PodSecurityPolicy` objects define a set of conditions that a pod must run with in order to be accepted into the system. Pod Security Policies are comprised of settings and strategies that control the security features a pod has access to and hence this must be used to control pod access permissions. This can be remediated by passing this argument as a value to the `enable-admission-plugins=` and pass that to `--kube-apiserver-arg=` argument to `k3s server`. An example can be found below.
@@ -432,16 +494,18 @@ This can be remediated by passing this argument as a value to the `--kube-apiser Ensure that the `--encryption-provider-config` argument is set as appropriate.
Rationale -Where `etcd` encryption is used, it is important to ensure that the appropriate set of encryption providers is used. Currently, the aescbc, kms and secretbox are likely to be appropriate options. +`etcd` is a highly available key-value store used by Kubernetes deployments for persistent storage of all of its REST API objects. These objects are sensitive in nature and should be encrypted at rest to avoid any disclosures. + +Detailed steps on how to configure secrets encryption in K3s are available in [Secrets Encryption](../secrets_encryption/).
### Control 1.2.34 Ensure that encryption providers are appropriately configured.
Rationale -`etcd` is a highly available key-value store used by Kubernetes deployments for persistent storage of all of its REST API objects. These objects are sensitive in nature and should be encrypted at rest to avoid any disclosures. +Where `etcd` encryption is used, it is important to ensure that the appropriate set of encryption providers is used. Currently, the `aescbc`, `kms` and `secretbox` are likely to be appropriate options. -This can be remediated by passing a valid configuration to `k3s` as outlined above. +This can be remediated by passing a valid configuration to `k3s` as outlined above. Detailed steps on how to configure secrets encryption in K3s are available in [Secrets Encryption](../secrets_encryption/).
### Control 1.3.1 @@ -454,7 +518,7 @@ This can be remediated by passing this argument as a value to the `--kube-apiser ### Control 3.2.1 -Ensure that a minimal audit policy is created (Scored) +Ensure that a minimal audit policy is created.
Rationale Logging is an important detective control for all systems, to detect potential unauthorized access. @@ -462,7 +526,6 @@ Logging is an important detective control for all systems, to detect potential u This can be remediated by passing controls 1.2.22 - 1.2.25 and verifying their efficacy.
- ### Control 4.2.7 Ensure that the `--make-iptables-util-chains` argument is set to true.
@@ -473,24 +536,23 @@ This can be remediated by passing this argument as a value to the `--kube-apiser
### Control 5.1.5 -Ensure that default service accounts are not actively used. (Scored) +Ensure that default service accounts are not actively used
Rationale - -Kubernetes provides a default service account which is used by cluster workloads where no specific service account is assigned to the pod. +Kubernetes provides a `default` service account which is used by cluster workloads where no specific service account is assigned to the pod. Where access to the Kubernetes API from a pod is required, a specific service account should be created for that pod, and rights granted to that service account. The default service account should be configured such that it does not provide a service account token and does not have any explicit rights assignments. -
-The remediation for this is to update the `automountServiceAccountToken` field to `false` for the `default` service account in each namespace. +This can be remediated by updating the `automountServiceAccountToken` field to `false` for the `default` service account in each namespace. For `default` service accounts in the built-in namespaces (`kube-system`, `kube-public`, `kube-node-lease`, and `default`), K3s does not automatically do this. You can manually update this field on these service accounts to pass the control. + ## Control Plane Execution and Arguments -Listed below are the K3s control plane components and the arguments they're given at start, by default. Commented to their right is the CIS 1.5 control that they satisfy. +Listed below are the K3s control plane components and the arguments they are given at start, by default. Commented to their right is the CIS 1.6 control that they satisfy. ```bash kube-apiserver @@ -590,13 +652,14 @@ kubelet --tls-private-key-file=/var/lib/rancher/k3s/agent/serving-kubelet.key # 4.2.10 ``` -The command below is an example of how the outlined remediations can be applied. +The command below is an example of how the outlined remediations can be applied to harden K3s. ```bash k3s server \ --protect-kernel-defaults=true \ --secrets-encryption=true \ - --kube-apiserver-arg='audit-log-path=/var/lib/rancher/k3s/server/logs/audit-log' \ + --kube-apiserver-arg='audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log' \ + --kube-apiserver-arg='audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml' \ --kube-apiserver-arg='audit-log-maxage=30' \ --kube-apiserver-arg='audit-log-maxbackup=10' \ --kube-apiserver-arg='audit-log-maxsize=100' \ @@ -611,4 +674,4 @@ k3s server \ ## Conclusion -If you have followed this guide, your K3s cluster will be configured to comply with the CIS Kubernetes Benchmark. You can review the [CIS Benchmark Self-Assessment Guide](../self_assessment/) to understand the expectations of each of the benchmarks and how you can do the same on your cluster. +If you have followed this guide, your K3s cluster will be configured to comply with the CIS Kubernetes Benchmark. You can review the [CIS Benchmark Self-Assessment Guide](../self_assessment/) to understand the expectations of each of the benchmark's checks and how you can do the same on your cluster. diff --git a/content/k3s/latest/en/security/secrets_encryption/_index.md b/content/k3s/latest/en/security/secrets_encryption/_index.md new file mode 100644 index 00000000000..a7491e2fb63 --- /dev/null +++ b/content/k3s/latest/en/security/secrets_encryption/_index.md @@ -0,0 +1,241 @@ +--- +title: Secrets Encryption +weight: 26 +--- + +# Secrets Encryption Config +_Available as of v1.17.4+k3s1_ + +K3s supports enabling secrets encryption at rest by passing the flag `--secrets-encryption` on a server; this flag will do the following automatically: + +- Generate an AES-CBC key +- Generate an encryption config file with the generated key +- Pass the config to the KubeAPI as encryption-provider-config + +Example of the encryption config file: +``` +{ + "kind": "EncryptionConfiguration", + "apiVersion": "apiserver.config.k8s.io/v1", + "resources": [ + { + "resources": [ + "secrets" + ], + "providers": [ + { + "aescbc": { + "keys": [ + { + "name": "aescbckey", + "secret": "xxxxxxxxxxxxxxxxxxx" + } + ] + } + }, + { + "identity": {} + } + ] + } + ] +} +``` + + +## Secrets Encryption Tool +_Available as of v1.21.8+k3s1_ + +K3s contains a utility tool `secrets-encrypt`, which enables automatic control over the following: + +- Disabling/Enabling secrets encryption +- Adding new encryption keys +- Rotating and deleting encryption keys +- Reencrypting secrets + +>**Warning:** Failure to follow proper procedure for rotating encryption keys can leave your cluster permanently corrupted. Proceed with caution. + +### Single-Server Encryption Key Rotation +To rotate secrets encryption keys on a single-node cluster: + +- Start the K3s server with the flag `--secrets-encryption` + +>**Note:** Starting K3s without encryption and enabling it at a later time is currently *not* supported. + +1. Prepare + + ``` + k3s secrets-encrypt prepare + ``` + +2. Kill and restart the K3s server with same arguments +3. Rotate + + ``` + k3s secrets-encrypt rotate + ``` + +4. Kill and restart the K3s server with same arguments +5. Reencrypt + + ``` + k3s secrets-encrypt reencrypt + ``` + +### High-Availability Encryption Key Rotation +The steps are the same for both embedded DB and external DB clusters. + +To rotate secrets encryption keys on HA setups: + +>**Notes:** +> +> - Starting K3s without encryption and enabling it at a later time is currently *not* supported. +> +> - While not required, it is recommended that you pick one server node from which to run the `secrets-encrypt` commands. + +- Start up all three K3s servers with the `--secrets-encryption` flag. For brevity, the servers will be referred to as S1, S2, S3. + +1. Prepare on S1 + + ``` + k3s secrets-encrypt prepare + ``` + +2. Kill and restart S1 with same arguments +3. Once S1 is up, kill and restart the S2 and S3 + +4. Rotate on S1 + + ``` + k3s secrets-encrypt rotate + ``` + +5. Kill and restart S1 with same arguments +6. Once S1 is up, kill and restart the S2 and S3 + +7. Reencrypt on S1 + + ``` + k3s secrets-encrypt reencrypt + ``` + +8. Kill and restart S1 with same arguments +9. Once S1 is up, kill and restart the S2 and S3 + +### Single-Server Secrets Encryption Disable/Enable +After launching a server with `--secrets-encryption` flag, secrets encryption can be disabled. + +To disable secrets encryption on a single-node cluster: + +1. Disable + + ``` + k3s secrets-encrypt disable + ``` + +2. Kill and restart the K3s server with same arguments + +3. Reencrypt with flags + + ``` + k3s secrets-encrypt reencrypt --force --skip + ``` + +To re-enable secrets encryption on a single node cluster: + +1. Enable + + ``` + k3s secrets-encrypt enable + ``` + +2. Kill and restart the K3s server with same arguments + +3. Reencrypt with flags + + ``` + k3s secrets-encrypt reencrypt --force --skip + ``` + +### High-Availability Secrets Encryption Disable/Enable +After launching a HA cluster with `--secrets-encryption` flags, secrets encryption can be disabled. +>**Note:** While not required, it is recommended that you pick one server node from which to run the `secrets-encrypt` commands. + +For brevity, the three servers used in this guide will be referred to as S1, S2, S3. + +To disable secrets encryption on a HA cluster: + +1. Disable on S1 + + ``` + k3s secrets-encrypt disable + ``` + +2. Kill and restart S1 with same arguments +3. Once S1 is up, kill and restart the S2 and S3 + + +4. Reencrypt with flags on S1 + + ``` + k3s secrets-encrypt reencrypt --force --skip + ``` + +To re-enable secrets encryption on a HA cluster: + +1. Enable on S1 + + ``` + k3s secrets-encrypt enable + ``` + +2. Kill and restart S1 with same arguments +3. Once S1 is up, kill and restart the S2 and S3 + +4. Reencrypt with flags on S1 + + ``` + k3s secrets-encrypt reencrypt --force --skip + ``` + + +### Secrets Encryption Status +The secrets-encrypt tool includes a `status` command that displays information about the current status of secrets encryption on the node. + +An example of the command on a single-server node: +``` +$ k3s secrets-encrypt status +Encryption Status: Enabled +Current Rotation Stage: start +Server Encryption Hashes: All hashes match + +Active Key Type Name +------ -------- ---- + * AES-CBC aescbckey + +``` + +Another example on HA cluster, after rotating the keys, but before restarting the servers: +``` +$ k3s secrets-encrypt status +Encryption Status: Enabled +Current Rotation Stage: rotate +Server Encryption Hashes: hash does not match between node-1 and node-2 + +Active Key Type Name +------ -------- ---- + * AES-CBC aescbckey-2021-12-10T22:54:38Z + AES-CBC aescbckey + +``` + +Details on each section are as follows: + +- __Encryption Status__: Displayed whether secrets encryption is disabled or enabled on the node +- __Current Rotation Stage__: Indicates the current rotation stage on the node. + Stages are: `start`, `prepare`, `rotate`, `reencrypt_request`, `reencrypt_active`, `reencrypt_finished` +- __Server Encryption Hashes__: Useful for HA clusters, this indicates whether all servers are on the same stage with their local files. This can be used to identify whether a restart of servers is required before proceeding to the next stage. In the HA example above, node-1 and node-2 have different hashes, indicating that they currently do not have the same encryption configuration. Restarting the servers will sync up their configuration. +- __Key Table__: Summarizes information about the secrets encryption keys found on the node. + * __Active__: The "*" indicates which, if any, of the keys are currently used for secrets encryption. An active key is used by Kubernetes to encrypt any new secrets. + * __Key Type__: All keys using this tool are `AES-CBC` type. See more info [here.](https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/#providers) + * __Name__: Name of the encryption key. \ No newline at end of file diff --git a/content/k3s/latest/en/security/self_assessment/_index.md b/content/k3s/latest/en/security/self_assessment/_index.md index ff7ba082384..6471a95fb83 100644 --- a/content/k3s/latest/en/security/self_assessment/_index.md +++ b/content/k3s/latest/en/security/self_assessment/_index.md @@ -1,18 +1,17 @@ --- -title: "CIS Self Assessment Guide" +title: CIS Self Assessment Guide weight: 90 --- - -### CIS Kubernetes Benchmark v1.5 - K3s v1.17, v1.18, & v1.19 +### CIS Kubernetes Benchmark v1.6 - K3s with Kubernetes v1.17 to v1.21 #### Overview -This document is a companion to the K3s security hardening guide. The hardening guide provides prescriptive guidance for hardening a production installation of K3s, and this benchmark guide is meant to help you evaluate the level of security of the hardened cluster against each control in the CIS Kubernetes benchmark. It is to be used by K3s operators, security teams, auditors, and decision-makers. +This document is a companion to the K3s security hardening guide. The hardening guide provides prescriptive guidance for hardening a production installation of K3s, and this benchmark guide is meant to help you evaluate the level of security of the hardened cluster against each control in the CIS Kubernetes Benchmark. It is to be used by K3s operators, security teams, auditors, and decision-makers. -This guide is specific to the **v1.17**, **v1.18**, and **v1.19** release line of K3s and the **v1.5.1** release of the CIS Kubernetes Benchmark. +This guide is specific to the **v1.17**, **v1.18**, **v1.19**, **v1.20** and **v1.21** release line of K3s and the **v1.6** release of the CIS Kubernetes Benchmark. -For more detail about each control, including more detailed descriptions and remediations for failing tests, you can refer to the corresponding section of the CIS Kubernetes Benchmark v1.5. You can download the benchmark after logging in to [CISecurity.org](https://www.cisecurity.org/benchmark/kubernetes/). +For more information about each control, including detailed descriptions and remediations for failing tests, you can refer to the corresponding section of the CIS Kubernetes Benchmark v1.6. You can download the benchmark, after creating a free account, in [Center for Internet Security (CIS)](https://www.cisecurity.org/benchmark/kubernetes/). #### Testing controls methodology @@ -24,2474 +23,3056 @@ These are the possible results for each control: - **Pass** - The K3s cluster under test passed the audit outlined in the benchmark. - **Not Applicable** - The control is not applicable to K3s because of how it is designed to operate. The remediation section will explain why this is so. -- **Not Scored - Operator Dependent** - The control is not scored in the CIS benchmark and it depends on the cluster's use case or some other factor that must be determined by the cluster operator. These controls have been evaluated to ensure K3s does not prevent their implementation, but no further configuration or auditing of the cluster under test has been performed. +- **Warn** - The control is manual in the CIS benchmark and it depends on the cluster's use case or some other factor that must be determined by the cluster operator. These controls have been evaluated to ensure K3s does not prevent their implementation, but no further configuration or auditing of the cluster under test has been performed. This guide makes the assumption that K3s is running as a Systemd unit. Your installation may vary and will require you to adjust the "audit" commands to fit your scenario. +> NOTE: Only `automated` tests (previously called `scored`) are covered in this guide. + ### Controls --- -## 1 Master Node Security Configuration -### 1.1 Master Node Configuration Files -#### 1.1.1 -Ensure that the API server pod specification file permissions are set to `644` or more restrictive (Scored) -
-Rationale -The API server pod specification file controls various parameters that set the behavior of the API server. You should restrict its file permissions to maintain the integrity of the file. The file should be writable by only the administrators on the system. -
+## 1.1 Master Node Configuration Files +### 1.1.1 Ensure that the API server pod specification file permissions are set to 644 or more restrictive (Automated) + **Result:** Not Applicable +**Remediation:** +Run the below command (based on the file location on your system) on the +master node. +For example, chmod 644 /etc/kubernetes/manifests/kube-apiserver.yaml + +### 1.1.2 Ensure that the API server pod specification file ownership is set to root:root (Automated) -#### 1.1.2 -Ensure that the API server pod specification file ownership is set to `root:root` (Scored) -
-Rationale -The API server pod specification file controls various parameters that set the behavior of the API server. You should set its file ownership to maintain the integrity of the file. The file should be owned by `root:root`. -
**Result:** Not Applicable +**Remediation:** +Run the below command (based on the file location on your system) on the master node. +For example, +chown root:root /etc/kubernetes/manifests/kube-apiserver.yaml + +### 1.1.3 Ensure that the controller manager pod specification file permissions are set to 644 or more restrictive (Automated) -#### 1.1.3 -Ensure that the controller manager pod specification file permissions are set to `644` or more restrictive (Scored) -
-Rationale -The controller manager pod specification file controls various parameters that set the behavior of the Controller Manager on the master node. You should restrict its file permissions to maintain the integrity of the file. The file should be writable by only the administrators on the system. -
**Result:** Not Applicable +**Remediation:** +Run the below command (based on the file location on your system) on the master node. +For example, +chmod 644 /etc/kubernetes/manifests/kube-controller-manager.yaml + +### 1.1.4 Ensure that the controller manager pod specification file ownership is set to root:root (Automated) -#### 1.1.4 -Ensure that the controller manager pod specification file ownership is set to `root:root` (Scored) -
-Rationale -The controller manager pod specification file controls various parameters that set the behavior of various components of the master node. You should set its file ownership to maintain the integrity of the file. The file should be owned by root:root. -
**Result:** Not Applicable +**Remediation:** +Run the below command (based on the file location on your system) on the master node. +For example, +chown root:root /etc/kubernetes/manifests/kube-controller-manager.yaml + +### 1.1.5 Ensure that the scheduler pod specification file permissions are set to 644 or more restrictive (Automated) -#### 1.1.5 -Ensure that the scheduler pod specification file permissions are set to `644` or more restrictive (Scored) -
-Rationale -The scheduler pod specification file controls various parameters that set the behavior of the Scheduler service in the master node. You should restrict its file permissions to maintain the integrity of the file. The file should be writable by only the administrators on the system. -
**Result:** Not Applicable +**Remediation:** +Run the below command (based on the file location on your system) on the master node. +For example, +chmod 644 /etc/kubernetes/manifests/kube-scheduler.yaml + +### 1.1.6 Ensure that the scheduler pod specification file ownership is set to root:root (Automated) -#### 1.1.6 -Ensure that the scheduler pod specification file ownership is set to `root:root` (Scored) -
-Rationale -The scheduler pod specification file controls various parameters that set the behavior of the kube-scheduler service in the master node. You should set its file ownership to maintain the integrity of the file. The file should be owned by root:root. -
**Result:** Not Applicable +**Remediation:** +Run the below command (based on the file location on your system) on the master node. +For example, +chown root:root /etc/kubernetes/manifests/kube-scheduler.yaml + +### 1.1.7 Ensure that the etcd pod specification file permissions are set to 644 or more restrictive (Automated) -#### 1.1.7 -Ensure that the etcd pod specification file permissions are set to `644` or more restrictive (Scored) -
-Rationale -The etcd pod specification file /var/lib/rancher/k3s/agent/pod-manifests/etcd.yaml controls various parameters that set the behavior of the etcd service in the master node. etcd is a highly- available key-value store which Kubernetes uses for persistent storage of all of its REST API object. You should restrict its file permissions to maintain the integrity of the file. The file should be writable by only the administrators on the system. -
**Result:** Not Applicable +**Remediation:** +Run the below command (based on the file location on your system) on the master node. +For example, +chmod 644 /etc/kubernetes/manifests/etcd.yaml + +### 1.1.8 Ensure that the etcd pod specification file ownership is set to root:root (Automated) -#### 1.1.8 -Ensure that the etcd pod specification file ownership is set to `root:root` (Scored) -
-Rationale -The etcd pod specification file /var/lib/rancher/k3s/agent/pod-manifests/etcd.yaml controls various parameters that set the behavior of the etcd service in the master node. etcd is a highly- available key-value store which Kubernetes uses for persistent storage of all of its REST API object. You should set its file ownership to maintain the integrity of the file. The file should be owned by root:root. -
**Result:** Not Applicable +**Remediation:** +Run the below command (based on the file location on your system) on the master node. +For example, +chown root:root /etc/kubernetes/manifests/etcd.yaml + +### 1.1.9 Ensure that the Container Network Interface file permissions are set to 644 or more restrictive (Manual) -#### 1.1.9 -Ensure that the Container Network Interface file permissions are set to 644 or more restrictive (Not Scored) -
-Rationale -Container Network Interface provides various networking options for overlay networking. You should consult their documentation and restrict their respective file permissions to maintain the integrity of those files. Those files should be writable by only the administrators on the system. -
**Result:** Not Applicable +**Remediation:** +Run the below command (based on the file location on your system) on the master node. +For example, +chmod 644 + +### 1.1.10 Ensure that the Container Network Interface file ownership is set to root:root (Manual) -#### 1.1.10 -Ensure that the Container Network Interface file ownership is set to root:root (Not Scored) -
-Rationale -Container Network Interface provides various networking options for overlay networking. You should consult their documentation and restrict their respective file permissions to maintain the integrity of those files. Those files should be owned by root:root. -
**Result:** Not Applicable +**Remediation:** +Run the below command (based on the file location on your system) on the master node. +For example, +chown root:root -#### 1.1.11 -Ensure that the etcd data directory permissions are set to 700 or more restrictive (Scored) -
-Rationale -etcd is a highly-available key-value store used by Kubernetes deployments for persistent storage of all of its REST API objects. This data directory should be protected from any unauthorized reads or writes. It should not be readable or writable by any group members or the world. -
+### 1.1.11 Ensure that the etcd data directory permissions are set to 700 or more restrictive (Automated) -**Result:** Pass -**Audit:** +**Result:** pass + +**Remediation:** +On the etcd server node, get the etcd data directory, passed as an argument --data-dir, +from the below command: +ps -ef | grep etcd +Run the below command (based on the etcd data directory found above). For example, +chmod 700 /var/lib/etcd + +**Audit Script:** `check_for_k3s_etcd.sh` + ```bash -stat -c %a /var/lib/rancher/k3s/server/db/etcd +#!/bin/bash + +# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) +# before it checks the requirement +set -eE + +handle_error() { + echo "false" +} + +trap 'handle_error' ERR + + +if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd' | grep -v grep | wc -l)" -gt 0 ]]; then + case $1 in + "1.1.11") + echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; + "1.2.29") + echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; + "2.1") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.2") + echo "$(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth')";; + "2.3") + echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.4") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.5") + echo "$(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth')";; + "2.6") + echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.7") + echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; + esac +else +# If another database is running, return whatever is required to pass the scan + case $1 in + "1.1.11") + echo "700";; + "1.2.29") + echo "--etcd-certfile AND --etcd-keyfile";; + "2.1") + echo "cert-file AND key-file";; + "2.2") + echo "true";; + "2.3") + echo "false";; + "2.4") + echo "peer-cert-file AND peer-key-file";; + "2.5") + echo "true";; + "2.6") + echo "--peer-auto-tls=false";; + "2.7") + echo "--trusted-ca-file";; + esac +fi + +``` + +**Audit Execution:** + +```bash +./check_for_k3s_etcd.sh 1.1.11 +``` + +**Expected Result**: + +```console +'700' is equal to '700' +``` + +**Returned Value**: + +```console 700 ``` -**Remediation:** -K3s manages the etcd data directory and sets its permissions to 700. No manual remediation needed. (only relevant when Etcd is used for the data store) +### 1.1.12 Ensure that the etcd data directory ownership is set to etcd:etcd (Automated) -#### 1.1.12 -Ensure that the etcd data directory ownership is set to `etcd:etcd` (Scored) -
-Rationale -etcd is a highly-available key-value store used by Kubernetes deployments for persistent storage of all of its REST API objects. This data directory should be protected from any unauthorized reads or writes. It should be owned by etcd:etcd. -
- **Result:** Not Applicable +**Remediation:** +On the etcd server node, get the etcd data directory, passed as an argument --data-dir, +from the below command: +ps -ef | grep etcd +Run the below command (based on the etcd data directory found above). +For example, chown etcd:etcd /var/lib/etcd -#### 1.1.13 -Ensure that the `admin.conf` file permissions are set to `644` or more restrictive (Scored) -
-Rationale -The admin.conf is the administrator kubeconfig file defining various settings for the administration of the cluster. You should restrict its file permissions to maintain the integrity of the file. The file should be writable by only the administrators on the system. +### 1.1.13 Ensure that the admin.conf file permissions are set to 644 or more restrictive (Automated) -In K3s, this file is located at `/var/lib/rancher/k3s/server/cred/admin.kubeconfig`. -
-**Result:** Pass +**Result:** Not Applicable **Remediation:** -By default, K3s creates the directory and files with the expected permissions of `644`. No manual remediation should be necessary. +Run the below command (based on the file location on your system) on the master node. +For example, +chmod 644 /var/lib/rancher/k3s/server/cred/admin.kubeconfig + +### 1.1.14 Ensure that the admin.conf file ownership is set to root:root (Automated) -#### 1.1.14 -Ensure that the `admin.conf` file ownership is set to `root:root` (Scored) -
-Rationale -The admin.conf file contains the admin credentials for the cluster. You should set its file ownership to maintain the integrity of the file. The file should be owned by root:root. - -In K3s, this file is located at `/var/lib/rancher/k3s/server/cred/admin.kubeconfig`. -
- -**Result:** Pass +**Result:** pass **Remediation:** -By default, K3s creates the directory and files with the expected ownership of `root:root`. No manual remediation should be necessary. - - -#### 1.1.15 -Ensure that the `scheduler.conf` file permissions are set to `644` or more restrictive (Scored) -
-Rationale - -The scheduler.conf file is the kubeconfig file for the Scheduler. You should restrict its file permissions to maintain the integrity of the file. The file should be writable by only the administrators on the system. - -In K3s, this file is located at `/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig`. -
- -**Result:** Pass - -**Remediation:** -By default, K3s creates the directory and files with the expected permissions of `644`. No manual remediation should be necessary. - - -#### 1.1.16 -Ensure that the `scheduler.conf` file ownership is set to `root:root` (Scored) -
-Rationale -The scheduler.conf file is the kubeconfig file for the Scheduler. You should set its file ownership to maintain the integrity of the file. The file should be owned by root:root. - -In K3s, this file is located at `/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig`. -
- -**Result:** Pass - -**Remediation:** -By default, K3s creates the directory and files with the expected ownership of `root:root`. No manual remediation should be necessary. - - -#### 1.1.17 -Ensure that the `controller.kubeconfig` file permissions are set to `644` or more restrictive (Scored) -
-Rationale -The controller.kubeconfig file is the kubeconfig file for the Scheduler. You should restrict its file permissions to maintain the integrity of the file. The file should be writable by only the administrators on the system. - -In K3s, this file is located at `/var/lib/rancher/k3s/server/cred/controller.kubeconfig`. -
- -**Result:** Pass - -**Remediation:** -By default, K3s creates the directory and files with the expected permissions of `644`. No manual remediation should be necessary. - - -#### 1.1.18 -Ensure that the `controller.kubeconfig` file ownership is set to `root:root` (Scored) -
-Rationale -The controller.kubeconfig file is the kubeconfig file for the Scheduler. You should set its file ownership to maintain the integrity of the file. The file should be owned by root:root. - -In K3s, this file is located at `/var/lib/rancher/k3s/server/cred/controller.kubeconfig`. -
- -**Result:** Pass - -**Remediation:** -By default, K3s creates the directory and files with the expected ownership of `root:root`. No manual remediation should be necessary. - - -#### 1.1.19 -Ensure that the Kubernetes PKI directory and file ownership is set to `root:root` (Scored) -
-Rationale -Kubernetes makes use of a number of certificates as part of its operation. You should set the ownership of the directory containing the PKI information and all files in that directory to maintain their integrity. The directory and files should be owned by root:root. -
- -**Result:** Pass +Run the below command (based on the file location on your system) on the master node. +For example, +chown root:root /etc/kubernetes/admin.conf **Audit:** + ```bash -stat -c %U:%G /var/lib/rancher/k3s/server/tls +/bin/sh -c 'if test -e /var/lib/rancher/k3s/server/cred/admin.kubeconfig; then stat -c %U:%G /var/lib/rancher/k3s/server/cred/admin.kubeconfig; fi' +``` + +**Expected Result**: + +```console +'root:root' is equal to 'root:root' +``` + +**Returned Value**: + +```console root:root ``` +### 1.1.15 Ensure that the scheduler.conf file permissions are set to 644 or more restrictive (Automated) + + +**Result:** pass + **Remediation:** -By default, K3s creates the directory and files with the expected ownership of `root:root`. No manual remediation should be necessary. - - -#### 1.1.20 -Ensure that the Kubernetes PKI certificate file permissions are set to `644` or more restrictive (Scored) -
-Rationale -Kubernetes makes use of a number of certificate files as part of the operation of its components. The permissions on these files should be set to 644 or more restrictive to protect their integrity. -
- -**Result:** Pass +Run the below command (based on the file location on your system) on the master node. +For example, +chmod 644 scheduler **Audit:** -Run the below command on the master node. ```bash -stat -c %n\ %a /var/lib/rancher/k3s/server/tls/*.crt +/bin/sh -c 'if test -e scheduler; then stat -c permissions=%a scheduler; fi' ``` -Verify that the permissions are `644` or more restrictive. +**Expected Result**: + +```console +'permissions' is not present +``` + +### 1.1.16 Ensure that the scheduler.conf file ownership is set to root:root (Automated) + + +**Result:** pass **Remediation:** -By default, K3s creates the files with the expected permissions of `644`. No manual remediation is needed. - - -#### 1.1.21 -Ensure that the Kubernetes PKI key file permissions are set to `600` (Scored) -
-Rationale -Kubernetes makes use of a number of key files as part of the operation of its components. The permissions on these files should be set to 600 to protect their integrity and confidentiality. -
- -**Result:** Pass +Run the below command (based on the file location on your system) on the master node. +For example, +chown root:root scheduler **Audit:** -Run the below command on the master node. ```bash -stat -c %n\ %a /var/lib/rancher/k3s/server/tls/*.key +/bin/sh -c 'if test -e scheduler; then stat -c %U:%G scheduler; fi' ``` -Verify that the permissions are `600` or more restrictive. +**Expected Result**: + +```console +'root:root' is not present +``` + +### 1.1.17 Ensure that the controller-manager.conf file permissions are set to 644 or more restrictive (Automated) + + +**Result:** pass **Remediation:** -By default, K3s creates the files with the expected permissions of `600`. No manual remediation is needed. - - -### 1.2 API Server -This section contains recommendations relating to API server configuration flags - - -#### 1.2.1 -Ensure that the `--anonymous-auth` argument is set to false (Not Scored) - -
-Rationale -When enabled, requests that are not rejected by other configured authentication methods are treated as anonymous requests. These requests are then served by the API server. You should rely on authentication to authorize access and disallow anonymous requests. - -If you are using RBAC authorization, it is generally considered reasonable to allow anonymous access to the API Server for health checks and discovery purposes, and hence this recommendation is not scored. However, you should consider whether anonymous discovery is an acceptable risk for your purposes. -
- -**Result:** Pass +Run the below command (based on the file location on your system) on the master node. +For example, +chmod 644 controllermanager **Audit:** -Run the below command on the master node. ```bash -journalctl -u k3s | grep "Running kube-apiserver" | tail -n1 | grep "anonymous-auth" +/bin/sh -c 'if test -e controllermanager; then stat -c permissions=%a controllermanager; fi' ``` -Verify that `--anonymous-auth=false` is present. +**Expected Result**: + +```console +'permissions' is not present +``` + +### 1.1.18 Ensure that the controller-manager.conf file ownership is set to root:root (Automated) + + +**Result:** pass **Remediation:** -By default, K3s kube-apiserver is configured to run with this flag and value. No manual remediation is needed. - -#### 1.2.2 -Ensure that the `--basic-auth-file` argument is not set (Scored) -
-Rationale -Basic authentication uses plaintext credentials for authentication. Currently, the basic authentication credentials last indefinitely, and the password cannot be changed without restarting the API server. The basic authentication is currently supported for convenience. Hence, basic authentication should not be used. -
- -**Result:** Pass +Run the below command (based on the file location on your system) on the master node. +For example, +chown root:root controllermanager **Audit:** -Run the below command on the master node. ```bash -journalctl -u k3s | grep "Running kube-apiserver" | tail -n1 | grep "basic-auth-file" +stat -c %U:%G /var/lib/rancher/k3s/server/tls ``` -Verify that the `--basic-auth-file` argument does not exist. +**Expected Result**: + +```console +'root:root' is equal to 'root:root' +``` + +**Returned Value**: + +```console +root:root +``` + +### 1.1.19 Ensure that the Kubernetes PKI directory and file ownership is set to root:root (Automated) + + +**Result:** pass **Remediation:** -By default, K3s does not run with basic authentication enabled. No manual remediation is needed. - - -#### 1.2.3 -Ensure that the `--token-auth-file` parameter is not set (Scored) - -
-Rationale -The token-based authentication utilizes static tokens to authenticate requests to the apiserver. The tokens are stored in clear-text in a file on the apiserver, and cannot be revoked or rotated without restarting the apiserver. Hence, do not use static token-based authentication. -
- -**Result:** Pass +Run the below command (based on the file location on your system) on the master node. +For example, +chown -R root:root /etc/kubernetes/pki/ **Audit:** -Run the below command on the master node. ```bash -journalctl -u k3s | grep "Running kube-apiserver" | tail -n1 | grep "token-auth-file" +find /etc/kubernetes/pki/ | xargs stat -c %U:%G ``` -Verify that the `--token-auth-file` argument does not exist. +**Expected Result**: + +```console +'root:root' is not present +``` + +### 1.1.20 Ensure that the Kubernetes PKI certificate file permissions are set to 644 or more restrictive (Manual) + + +**Result:** pass **Remediation:** -By default, K3s does not run with basic authentication enabled. No manual remediation is needed. +Run the below command (based on the file location on your system) on the master node. +For example, +chmod -R 644 /etc/kubernetes/pki/*.crt -#### 1.2.4 -Ensure that the `--kubelet-https` argument is set to true (Scored) +**Audit:** + +```bash +stat -c %n %a /var/lib/rancher/k3s/server/tls/*.crt +``` + +**Expected Result**: + +```console +'permissions' is not present +``` + +### 1.1.21 Ensure that the Kubernetes PKI key file permissions are set to 600 (Manual) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the master node. +For example, +chmod -R 600 /etc/kubernetes/pki/*.key + +**Audit:** + +```bash +stat -c %n %a /var/lib/rancher/k3s/server/tls/*.key +``` + +**Expected Result**: + +```console +'permissions' is not present +``` + +## 1.2 API Server +### 1.2.1 Ensure that the --anonymous-auth argument is set to false (Manual) + + +**Result:** warn + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the master node and set the below parameter. +--anonymous-auth=false + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'anonymous-auth' +``` + +### 1.2.2 Ensure that the --basic-auth-file argument is not set (Automated) + + +**Result:** pass + +**Remediation:** +Follow the documentation and configure alternate mechanisms for authentication. Then, +edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the master node and remove the --basic-auth-file= parameter. + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'basic-auth-file' +``` + +**Expected Result**: + +```console +'--basic-auth-file' is not present +``` + +### 1.2.3 Ensure that the --token-auth-file parameter is not set (Automated) + + +**Result:** pass + +**Remediation:** +Follow the documentation and configure alternate mechanisms for authentication. Then, +edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the master node and remove the --token-auth-file= parameter. + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'token-auth-file' +``` + +**Expected Result**: + +```console +'--token-auth-file' is not present +``` + +### 1.2.4 Ensure that the --kubelet-https argument is set to true (Automated) -
-Rationale -Connections from apiserver to kubelets could potentially carry sensitive data such as secrets and keys. It is thus important to use in-transit encryption for any communication between the apiserver and kubelets. -
**Result:** Not Applicable +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the master node and remove the --kubelet-https parameter. + +### 1.2.5 Ensure that the --kubelet-client-certificate and --kubelet-client-key arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection between the +apiserver and kubelets. Then, edit API server pod specification file +/etc/kubernetes/manifests/kube-apiserver.yaml on the master node and set the +kubelet client certificate and key parameters as below. +--kubelet-client-certificate= +--kubelet-client-key= + **Audit:** -Run the below command on the master node. ```bash -journalctl -u k3s | grep "Running kube-apiserver" | tail -n1 | grep "kubelet-https" +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'kubelet-certificate-authority' ``` -Verify that the `--kubelet-https` argument does not exist. +**Expected Result**: + +```console +'--kubelet-client-certificate' is not present AND '--kubelet-client-key' is not present +``` + +### 1.2.6 Ensure that the --kubelet-certificate-authority argument is set as appropriate (Automated) + + +**Result:** pass **Remediation:** -By default, K3s kube-apiserver doesn't run with the `--kubelet-https` parameter as it runs with TLS. No manual remediation is needed. - -#### 1.2.5 -Ensure that the `--kubelet-client-certificate` and `--kubelet-client-key` arguments are set as appropriate (Scored) - -
-Rationale -The apiserver, by default, does not authenticate itself to the kubelet's HTTPS endpoints. The requests from the apiserver are treated anonymously. You should set up certificate-based kubelet authentication to ensure that the apiserver authenticates itself to kubelets when submitting requests. -
- -**Result:** Pass +Follow the Kubernetes documentation and setup the TLS connection between +the apiserver and kubelets. Then, edit the API server pod specification file +/etc/kubernetes/manifests/kube-apiserver.yaml on the master node and set the +--kubelet-certificate-authority parameter to the path to the cert file for the certificate authority. +--kubelet-certificate-authority= **Audit:** -Run the below command on the master node. ```bash -journalctl -u k3s | grep "Running kube-apiserver" | tail -n1 | grep -E 'kubelet-client-certificate|kubelet-client-key' +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'kubelet-certificate-authority' ``` -Verify that the `--kubelet-client-certificate` and `--kubelet-client-key` arguments exist and they are set as appropriate. +**Expected Result**: + +```console +'--kubelet-certificate-authority' is not present +``` + +### 1.2.7 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated) + + +**Result:** pass **Remediation:** -By default, K3s kube-apiserver is ran with these arguments for secure communication with kubelet. No manual remediation is needed. - - -#### 1.2.6 -Ensure that the `--kubelet-certificate-authority` argument is set as appropriate (Scored) -
-Rationale -The connections from the apiserver to the kubelet are used for fetching logs for pods, attaching (through kubectl) to running pods, and using the kubelet’s port-forwarding functionality. These connections terminate at the kubelet’s HTTPS endpoint. By default, the apiserver does not verify the kubelet’s serving certificate, which makes the connection subject to man-in-the-middle attacks, and unsafe to run over untrusted and/or public networks. -
- -**Result:** Pass +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the master node and set the --authorization-mode parameter to values other than AlwaysAllow. +One such example could be as below. +--authorization-mode=RBAC **Audit:** -Run the below command on the master node. ```bash -journalctl -u k3s | grep "Running kube-apiserver" | tail -n1 | grep "kubelet-certificate-authority" +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'authorization-mode' ``` -Verify that the `--kubelet-certificate-authority` argument exists and is set as appropriate. +**Expected Result**: + +```console +'--authorization-mode' is not present +``` + +### 1.2.8 Ensure that the --authorization-mode argument includes Node (Automated) + + +**Result:** pass **Remediation:** -By default, K3s kube-apiserver is ran with this argument for secure communication with kubelet. No manual remediation is needed. - - -#### 1.2.7 -Ensure that the `--authorization-mode` argument is not set to `AlwaysAllow` (Scored) -
-Rationale -The API Server, can be configured to allow all requests. This mode should not be used on any production cluster. -
- -**Result:** Pass +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the master node and set the --authorization-mode parameter to a value that includes Node. +--authorization-mode=Node,RBAC **Audit:** -Run the below command on the master node. ```bash -journalctl -u k3s | grep "Running kube-apiserver" | tail -n1 | grep "authorization-mode" +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'authorization-mode' ``` -Verify that the argument value doesn't contain `AlwaysAllow`. +**Expected Result**: + +```console +'--authorization-mode' is not present +``` + +### 1.2.9 Ensure that the --authorization-mode argument includes RBAC (Automated) + + +**Result:** pass **Remediation:** -By default, K3s sets `Node,RBAC` as the parameter to the `--authorization-mode` argument. No manual remediation is needed. - - -#### 1.2.8 -Ensure that the `--authorization-mode` argument includes `Node` (Scored) -
-Rationale -The Node authorization mode only allows kubelets to read Secret, ConfigMap, PersistentVolume, and PersistentVolumeClaim objects associated with their nodes. -
- -**Result:** Pass +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the master node and set the --authorization-mode parameter to a value that includes RBAC, +for example: +--authorization-mode=Node,RBAC **Audit:** -Run the below command on the master node. ```bash -journalctl -u k3s | grep "Running kube-apiserver" | tail -n1 | grep "authorization-mode" +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'authorization-mode' ``` -Verify `Node` exists as a parameter to the argument. +**Expected Result**: + +```console +'--authorization-mode' is not present +``` + +### 1.2.10 Ensure that the admission control plugin EventRateLimit is set (Manual) + + +**Result:** pass **Remediation:** -By default, K3s sets `Node,RBAC` as the parameter to the `--authorization-mode` argument. No manual remediation is needed. - - -#### 1.2.9 -Ensure that the `--authorization-mode` argument includes `RBAC` (Scored) -
-Rationale -Role Based Access Control (RBAC) allows fine-grained control over the operations that different entities can perform on different objects in the cluster. It is recommended to use the RBAC authorization mode. -
- -**Result:** Pass +Follow the Kubernetes documentation and set the desired limits in a configuration file. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +and set the below parameters. +--enable-admission-plugins=...,EventRateLimit,... +--admission-control-config-file= **Audit:** -Run the below command on the master node. ```bash -journalctl -u k3s | grep "Running kube-apiserver" | tail -n1 | grep "authorization-mode" +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'enable-admission-plugins' ``` -Verify `RBAC` exists as a parameter to the argument. +**Expected Result**: + +```console +'--enable-admission-plugins' is not present +``` + +### 1.2.11 Ensure that the admission control plugin AlwaysAdmit is not set (Automated) + + +**Result:** pass **Remediation:** -By default, K3s sets `Node,RBAC` as the parameter to the `--authorization-mode` argument. No manual remediation is needed. - - -#### 1.2.10 -Ensure that the admission control plugin EventRateLimit is set (Not Scored) -
-Rationale -Using `EventRateLimit` admission control enforces a limit on the number of events that the API Server will accept in a given time slice. A misbehaving workload could overwhelm and DoS the API Server, making it unavailable. This particularly applies to a multi-tenant cluster, where there might be a small percentage of misbehaving tenants which could have a significant impact on the performance of the cluster overall. Hence, it is recommended to limit the rate of events that the API server will accept. - -Note: This is an Alpha feature in the Kubernetes 1.15 release. -
- -**Result:** **Not Scored - Operator Dependent** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the master node and either remove the --enable-admission-plugins parameter, or set it to a +value that does not include AlwaysAdmit. **Audit:** -Run the below command on the master node. ```bash -journalctl -u k3s | grep "Running kube-apiserver" | tail -n1 | grep "enable-admission-plugins" +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'enable-admission-plugins' ``` -Verify that the `--enable-admission-plugins` argument is set to a value that includes EventRateLimit. +**Expected Result**: + +```console +'--enable-admission-plugins' is not present OR '--enable-admission-plugins' is not present +``` + +### 1.2.12 Ensure that the admission control plugin AlwaysPullImages is set (Manual) + + +**Result:** pass **Remediation:** -By default, K3s only sets `NodeRestriction,PodSecurityPolicy` as the parameter to the `--enable-admission-plugins` argument. -To configure this, follow the Kubernetes documentation and set the desired limits in a configuration file. Then refer to K3s's documentation to see how to supply additional api server configuration via the kube-apiserver-arg parameter. - - -#### 1.2.11 -Ensure that the admission control plugin `AlwaysAdmit` is not set (Scored) -
-Rationale -Setting admission control plugin AlwaysAdmit allows all requests and do not filter any requests. - -The AlwaysAdmit admission controller was deprecated in Kubernetes v1.13. Its behavior was equivalent to turning off all admission controllers. -
- -**Result:** Pass +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the master node and set the --enable-admission-plugins parameter to include +AlwaysPullImages. +--enable-admission-plugins=...,AlwaysPullImages,... **Audit:** -Run the below command on the master node. ```bash -journalctl -u k3s | grep "Running kube-apiserver" | tail -n1 | grep "enable-admission-plugins" +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'enable-admission-plugins' ``` -Verify that if the `--enable-admission-plugins` argument is set, its value does not include `AlwaysAdmit`. +**Expected Result**: + +```console +'--enable-admission-plugins' is not present +``` + +### 1.2.13 Ensure that the admission control plugin SecurityContextDeny is set if PodSecurityPolicy is not used (Manual) + + +**Result:** pass **Remediation:** -By default, K3s only sets `NodeRestriction,PodSecurityPolicy` as the parameter to the `--enable-admission-plugins` argument. No manual remediation needed. - - -#### 1.2.12 -Ensure that the admission control plugin AlwaysPullImages is set (Not Scored) -
-Rationale -Setting admission control policy to `AlwaysPullImages` forces every new pod to pull the required images every time. In a multi-tenant cluster users can be assured that their private images can only be used by those who have the credentials to pull them. Without this admission control policy, once an image has been pulled to a node, any pod from any user can use it simply by knowing the image’s name, without any authorization check against the image ownership. When this plug-in is enabled, images are always pulled prior to starting containers, which means valid credentials are required. - -
- -**Result:** **Not Scored - Operator Dependent** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the master node and set the --enable-admission-plugins parameter to include +SecurityContextDeny, unless PodSecurityPolicy is already in place. +--enable-admission-plugins=...,SecurityContextDeny,... **Audit:** -Run the below command on the master node. ```bash -journalctl -u k3s | grep "Running kube-apiserver" | tail -n1 | grep "enable-admission-plugins" +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'enable-admission-plugins' ``` -Verify that the `--enable-admission-plugins` argument is set to a value that includes `AlwaysPullImages`. +**Expected Result**: + +```console +'--enable-admission-plugins' is not present OR '--enable-admission-plugins' is not present +``` + +### 1.2.14 Ensure that the admission control plugin ServiceAccount is set (Automated) + + +**Result:** pass **Remediation:** -By default, K3s only sets `NodeRestriction,PodSecurityPolicy` as the parameter to the `--enable-admission-plugins` argument. -To configure this, follow the Kubernetes documentation and set the desired limits in a configuration file. Then refer to K3s's documentation to see how to supply additional api server configuration via the kube-apiserver-arg parameter. - -#### 1.2.13 -Ensure that the admission control plugin SecurityContextDeny is set if PodSecurityPolicy is not used (Not Scored) -
-Rationale -SecurityContextDeny can be used to provide a layer of security for clusters which do not have PodSecurityPolicies enabled. -
- -**Result:** Not Scored +Follow the documentation and create ServiceAccount objects as per your environment. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the master node and ensure that the --disable-admission-plugins parameter is set to a +value that does not include ServiceAccount. **Audit:** -Run the below command on the master node. ```bash -journalctl -u k3s | grep "Running kube-apiserver" | tail -n1 | grep "enable-admission-plugins" +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'ServiceAccount' ``` -Verify that the `--enable-admission-plugins` argument is set to a value that includes `SecurityContextDeny`, if `PodSecurityPolicy` is not included. +**Expected Result**: + +```console +'--disable-admission-plugins' is not present OR '--disable-admission-plugins' is not present +``` + +### 1.2.15 Ensure that the admission control plugin NamespaceLifecycle is set (Automated) + + +**Result:** pass **Remediation:** -K3s would need to have the `SecurityContextDeny` admission plugin enabled by passing it as an argument to K3s. `--kube-apiserver-arg='enable-admission-plugins=SecurityContextDeny` - - -#### 1.2.14 -Ensure that the admission control plugin `ServiceAccount` is set (Scored) -
-Rationale -When you create a pod, if you do not specify a service account, it is automatically assigned the `default` service account in the same namespace. You should create your own service account and let the API server manage its security tokens. -
- -**Result:** Pass +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the master node and set the --disable-admission-plugins parameter to +ensure it does not include NamespaceLifecycle. **Audit:** -Run the below command on the master node. ```bash -journalctl -u k3s | grep "Running kube-apiserver" | tail -n1 | grep "ServiceAccount" +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'disable-admission-plugins' ``` -Verify that the `--disable-admission-plugins` argument is set to a value that does not includes `ServiceAccount`. +**Expected Result**: + +```console +'--disable-admission-plugins' is not present OR '--disable-admission-plugins' is not present +``` + +### 1.2.16 Ensure that the admission control plugin PodSecurityPolicy is set (Automated) + + +**Result:** pass **Remediation:** -By default, K3s does not use this argument. If there's a desire to use this argument, follow the documentation and create ServiceAccount objects as per your environment. Then refer to K3s's documentation to see how to supply additional api server configuration via the kube-apiserver-arg parameter. - - -#### 1.2.15 -Ensure that the admission control plugin `NamespaceLifecycle` is set (Scored) -
-Rationale -Setting admission control policy to `NamespaceLifecycle` ensures that objects cannot be created in non-existent namespaces, and that namespaces undergoing termination are not used for creating the new objects. This is recommended to enforce the integrity of the namespace termination process and also for the availability of the newer objects. -
- -**Result:** Pass +Follow the documentation and create Pod Security Policy objects as per your environment. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the master node and set the --enable-admission-plugins parameter to a +value that includes PodSecurityPolicy: +--enable-admission-plugins=...,PodSecurityPolicy,... +Then restart the API Server. **Audit:** -Run the below command on the master node. ```bash -journalctl -u k3s | grep "Running kube-apiserver" | tail -n1 | grep "disable-admission-plugins" +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'enable-admission-plugins' ``` -Verify that the `--disable-admission-plugins` argument is set to a value that does not include `NamespaceLifecycle`. +**Expected Result**: + +```console +'--enable-admission-plugins' is not present +``` + +### 1.2.17 Ensure that the admission control plugin NodeRestriction is set (Automated) + + +**Result:** pass **Remediation:** -By default, K3s does not use this argument. No manual remediation needed. - - -#### 1.2.16 -Ensure that the admission control plugin `PodSecurityPolicy` is set (Scored) -
-Rationale -A Pod Security Policy is a cluster-level resource that controls the actions that a pod can perform and what it has the ability to access. The `PodSecurityPolicy` objects define a set of conditions that a pod must run with in order to be accepted into the system. Pod Security Policies are comprised of settings and strategies that control the security features a pod has access to and hence this must be used to control pod access permissions. - -**Note:** When the PodSecurityPolicy admission plugin is in use, there needs to be at least one PodSecurityPolicy in place for ANY pods to be admitted. See section 1.7 for recommendations on PodSecurityPolicy settings. -
- -**Result:** Pass +Follow the Kubernetes documentation and configure NodeRestriction plug-in on kubelets. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the master node and set the --enable-admission-plugins parameter to a +value that includes NodeRestriction. +--enable-admission-plugins=...,NodeRestriction,... **Audit:** -Run the below command on the master node. ```bash -journalctl -u k3s | grep "Running kube-apiserver" | tail -n1 | grep "enable-admission-plugins" +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'enable-admission-plugins' ``` -Verify that the `--enable-admission-plugins` argument is set to a value that includes `PodSecurityPolicy`. +**Expected Result**: + +```console +'--enable-admission-plugins' is not present +``` + +### 1.2.18 Ensure that the --insecure-bind-address argument is not set (Automated) + + +**Result:** pass **Remediation:** -K3s would need to have the `PodSecurityPolicy` admission plugin enabled by passing it as an argument to K3s. `--kube-apiserver-arg='enable-admission-plugins=PodSecurityPolicy`. - - -#### 1.2.17 -Ensure that the admission control plugin `NodeRestriction` is set (Scored) -
-Rationale -Using the `NodeRestriction` plug-in ensures that the kubelet is restricted to the `Node` and `Pod` objects that it could modify as defined. Such kubelets will only be allowed to modify their own `Node` API object, and only modify `Pod` API objects that are bound to their node. - -
- -**Result:** Pass +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the master node and remove the --insecure-bind-address parameter. **Audit:** -Run the below command on the master node. ```bash -journalctl -u k3s | grep "Running kube-apiserver" | tail -n1 | grep "enable-admission-plugins" +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'insecure-bind-address' ``` -Verify that the `--enable-admission-plugins` argument is set to a value that includes `NodeRestriction`. +**Expected Result**: + +```console +'--insecure-bind-address' is not present +``` + +### 1.2.19 Ensure that the --insecure-port argument is set to 0 (Automated) + + +**Result:** pass **Remediation:** -K3s would need to have the `NodeRestriction` admission plugin enabled by passing it as an argument to K3s. `--kube-apiserver-arg='enable-admission-plugins=NodeRestriction`. - - -#### 1.2.18 -Ensure that the `--insecure-bind-address` argument is not set (Scored) -
-Rationale -If you bind the apiserver to an insecure address, basically anyone who could connect to it over the insecure port, would have unauthenticated and unencrypted access to your master node. The apiserver doesn't do any authentication checking for insecure binds and traffic to the Insecure API port is not encrpyted, allowing attackers to potentially read sensitive data in transit. -
- -**Result:** Pass +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the master node and set the below parameter. +--insecure-port=0 **Audit:** -Run the below command on the master node. ```bash -journalctl -u k3s | grep "Running kube-apiserver" | tail -n1 | grep "insecure-bind-address" +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'insecure-port' ``` -Verify that the `--insecure-bind-address` argument does not exist. +**Expected Result**: + +```console +'--insecure-port' is not present +``` + +### 1.2.20 Ensure that the --secure-port argument is not set to 0 (Automated) + + +**Result:** pass **Remediation:** -By default, K3s explicitly excludes the use of the `--insecure-bind-address` parameter. No manual remediation is needed. - - -#### 1.2.19 -Ensure that the `--insecure-port` argument is set to `0` (Scored) -
-Rationale -Setting up the apiserver to serve on an insecure port would allow unauthenticated and unencrypted access to your master node. This would allow attackers who could access this port, to easily take control of the cluster. -
- -**Result:** Pass +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the master node and either remove the --secure-port parameter or +set it to a different (non-zero) desired port. **Audit:** -Run the below command on the master node. ```bash -journalctl -u k3s | grep "Running kube-apiserver" | tail -n1 | grep "insecure-port" +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'secure-port' ``` -Verify that the `--insecure-port` argument is set to `0`. +**Expected Result**: + +```console +'--secure-port' is not present OR '--secure-port' is not present +``` + +### 1.2.21 Ensure that the --profiling argument is set to false (Automated) + + +**Result:** pass **Remediation:** -By default, K3s starts the kube-apiserver process with this argument's parameter set to `0`. No manual remediation is needed. - - -#### 1.2.20 -Ensure that the `--secure-port` argument is not set to `0` (Scored) -
-Rationale -The secure port is used to serve https with authentication and authorization. If you disable it, no https traffic is served and all traffic is served unencrypted. -
- -**Result:** Pass +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the master node and set the below parameter. +--profiling=false **Audit:** -Run the below command on the master node. ```bash -journalctl -u k3s | grep "Running kube-apiserver" | tail -n1 | grep "secure-port" +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'profiling' ``` -Verify that the `--secure-port` argument is either not set or is set to an integer value between 1 and 65535. +**Expected Result**: + +```console +'--profiling' is not present +``` + +### 1.2.22 Ensure that the --audit-log-path argument is set (Automated) + + +**Result:** pass **Remediation:** -By default, K3s sets the parameter of 6444 for the `--secure-port` argument. No manual remediation is needed. - - -#### 1.2.21 -Ensure that the `--profiling` argument is set to `false` (Scored) -
-Rationale -Profiling allows for the identification of specific performance bottlenecks. It generates a significant amount of program data that could potentially be exploited to uncover system and program details. If you are not experiencing any bottlenecks and do not need the profiler for troubleshooting purposes, it is recommended to turn it off to reduce the potential attack surface. -
- -**Result:** Pass +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the master node and set the --audit-log-path parameter to a suitable path and +file where you would like audit logs to be written, for example: +--audit-log-path=/var/log/apiserver/audit.log **Audit:** -Run the below command on the master node. ```bash -journalctl -u k3s | grep "Running kube-apiserver" | tail -n1 | grep "profiling" +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'audit-log-path' ``` -Verify that the `--profiling` argument is set to false. +**Expected Result**: + +```console +'--audit-log-path' is not present +``` + +### 1.2.23 Ensure that the --audit-log-maxage argument is set to 30 or as appropriate (Automated) + + +**Result:** pass **Remediation:** -By default, K3s sets the `--profiling` flag parameter to false. No manual remediation needed. - - -#### 1.2.22 -Ensure that the `--audit-log-path` argument is set (Scored) -
-Rationale -Auditing the Kubernetes API Server provides a security-relevant chronological set of records documenting the sequence of activities that have affected system by individual users, administrators or other components of the system. Even though currently, Kubernetes provides only basic audit capabilities, it should be enabled. You can enable it by setting an appropriate audit log path. -
- -**Result:** Pass +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the master node and set the --audit-log-maxage parameter to 30 or as an appropriate number of days: +--audit-log-maxage=30 **Audit:** -Run the below command on the master node. ```bash -journalctl -u k3s | grep "Running kube-apiserver" | tail -n1 | grep "audit-log-path" +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'audit-log-maxage' ``` -Verify that the `--audit-log-path` argument is set as appropriate. +**Expected Result**: + +```console +'--audit-log-maxage' is not present +``` + +### 1.2.24 Ensure that the --audit-log-maxbackup argument is set to 10 or as appropriate (Automated) + + +**Result:** pass **Remediation:** -K3s server needs to be run with the following argument, `--kube-apiserver-arg='audit-log-path=/path/to/log/file'`. - - -#### 1.2.23 -Ensure that the `--audit-log-maxage` argument is set to `30` or as appropriate (Scored) -
-Rationale -Retaining logs for at least 30 days ensures that you can go back in time and investigate or correlate any events. Set your audit log retention period to 30 days or as per your business requirements. -
- -**Result:** Pass +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the master node and set the --audit-log-maxbackup parameter to 10 or to an appropriate +value. +--audit-log-maxbackup=10 **Audit:** -Run the below command on the master node. ```bash -journalctl -u k3s | grep "Running kube-apiserver" | tail -n1 | grep "audit-log-maxage" +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'audit-log-maxbackup' ``` -Verify that the `--audit-log-maxage` argument is set to `30` or as appropriate. +**Expected Result**: + +```console +'--audit-log-maxbackup' is not present +``` + +### 1.2.25 Ensure that the --audit-log-maxsize argument is set to 100 or as appropriate (Automated) + + +**Result:** pass **Remediation:** -K3s server needs to be run with the following argument, `--kube-apiserver-arg='audit-log-maxage=30'`. - - -#### 1.2.24 -Ensure that the `--audit-log-maxbackup` argument is set to `10` or as appropriate (Scored) -
-Rationale -Kubernetes automatically rotates the log files. Retaining old log files ensures that you would have sufficient log data available for carrying out any investigation or correlation. For example, if you have set file size of 100 MB and the number of old log files to keep as 10, you would approximate have 1 GB of log data that you could potentially use for your analysis. -
- -**Result:** Pass +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the master node and set the --audit-log-maxsize parameter to an appropriate size in MB. +For example, to set it as 100 MB: +--audit-log-maxsize=100 **Audit:** -Run the below command on the master node. ```bash -journalctl -u k3s | grep "Running kube-apiserver" | tail -n1 | grep "audit-log-maxbackup" +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'audit-log-maxsize' ``` -Verify that the `--audit-log-maxbackup` argument is set to `10` or as appropriate. +**Expected Result**: + +```console +'--audit-log-maxsize' is not present +``` + +### 1.2.26 Ensure that the --request-timeout argument is set as appropriate (Automated) + + +**Result:** pass **Remediation:** -K3s server needs to be run with the following argument, `--kube-apiserver-arg='audit-log-maxbackup=10'`. - - -#### 1.2.25 -Ensure that the `--audit-log-maxsize` argument is set to `100` or as appropriate (Scored) -
-Rationale -Kubernetes automatically rotates the log files. Retaining old log files ensures that you would have sufficient log data available for carrying out any investigation or correlation. If you have set file size of 100 MB and the number of old log files to keep as 10, you would approximate have 1 GB of log data that you could potentially use for your analysis. -
- -**Result:** Pass +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +and set the below parameter as appropriate and if needed. +For example, +--request-timeout=300s **Audit:** -Run the below command on the master node. ```bash -journalctl -u k3s | grep "Running kube-apiserver" | tail -n1 | grep "audit-log-maxsize" +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'request-timeout' ``` -Verify that the `--audit-log-maxsize` argument is set to `100` or as appropriate. +**Expected Result**: + +```console +'--request-timeout' is not present OR '--request-timeout' is not present +``` + +### 1.2.27 Ensure that the --service-account-lookup argument is set to true (Automated) + + +**Result:** pass **Remediation:** -K3s server needs to be run with the following argument, `--kube-apiserver-arg='audit-log-maxsize=100'`. - - -#### 1.2.26 -Ensure that the `--request-timeout` argument is set as appropriate (Scored) -
-Rationale -Setting global request timeout allows extending the API server request timeout limit to a duration appropriate to the user's connection speed. By default, it is set to 60 seconds which might be problematic on slower connections making cluster resources inaccessible once the data volume for requests exceeds what can be transmitted in 60 seconds. But, setting this timeout limit to be too large can exhaust the API server resources making it prone to Denial-of-Service attack. Hence, it is recommended to set this limit as appropriate and change the default limit of 60 seconds only if needed. -
- -**Result:** Pass +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the master node and set the below parameter. +--service-account-lookup=true +Alternatively, you can delete the --service-account-lookup parameter from this file so +that the default takes effect. **Audit:** -Run the below command on the master node. ```bash -journalctl -u k3s | grep "Running kube-apiserver" | tail -n1 | grep "request-timeout" +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'service-account-lookup' ``` -Verify that the `--request-timeout` argument is either not set or set to an appropriate value. +**Expected Result**: + +```console +'--service-account-lookup' is not present OR '--service-account-lookup' is not present +``` + +### 1.2.28 Ensure that the --service-account-key-file argument is set as appropriate (Automated) + + +**Result:** pass **Remediation:** -By default, K3s does not set the `--request-timeout` argument. No manual remediation needed. - - -#### 1.2.27 -Ensure that the `--service-account-lookup` argument is set to `true` (Scored) -
-Rationale -If `--service-account-lookup` is not enabled, the apiserver only verifies that the authentication token is valid, and does not validate that the service account token mentioned in the request is actually present in etcd. This allows using a service account token even after the corresponding service account is deleted. This is an example of time of check to time of use security issue. -
- -**Result:** Pass +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the master node and set the --service-account-key-file parameter +to the public key file for service accounts: +--service-account-key-file= **Audit:** -Run the below command on the master node. ```bash -journalctl -u k3s | grep "Running kube-apiserver" | tail -n1 | grep "service-account-lookup" +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'service-account-key-file' ``` -Verify that if the `--service-account-lookup` argument exists it is set to `true`. +**Expected Result**: + +```console +'--service-account-key-file' is not present +``` + +### 1.2.29 Ensure that the --etcd-certfile and --etcd-keyfile arguments are set as appropriate (Automated) + + +**Result:** pass **Remediation:** -K3s server needs to be run with the following argument, `--kube-apiserver-arg='service-account-lookup=true'`. +Follow the Kubernetes documentation and set up the TLS connection between the apiserver and etcd. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the master node and set the etcd certificate and key file parameters. +--etcd-certfile= +--etcd-keyfile= - -#### 1.2.28 -Ensure that the `--service-account-key-file` argument is set as appropriate (Scored) -
-Rationale -By default, if no `--service-account-key-file` is specified to the apiserver, it uses the private key from the TLS serving certificate to verify service account tokens. To ensure that the keys for service account tokens could be rotated as needed, a separate public/private key pair should be used for signing service account tokens. Hence, the public key should be specified to the apiserver with `--service-account-key-file`. -
- -**Result:** Pass - -**Audit:** -Run the below command on the master node. +**Audit Script:** `check_for_k3s_etcd.sh` ```bash -journalctl -u k3s | grep "Running kube-apiserver" | tail -n1 | grep "service-account-key-file" +#!/bin/bash + +# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) +# before it checks the requirement +set -eE + +handle_error() { + echo "false" +} + +trap 'handle_error' ERR + + +if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd' | grep -v grep | wc -l)" -gt 0 ]]; then + case $1 in + "1.1.11") + echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; + "1.2.29") + echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; + "2.1") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.2") + echo "$(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth')";; + "2.3") + echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.4") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.5") + echo "$(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth')";; + "2.6") + echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.7") + echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; + esac +else +# If another database is running, return whatever is required to pass the scan + case $1 in + "1.1.11") + echo "700";; + "1.2.29") + echo "--etcd-certfile AND --etcd-keyfile";; + "2.1") + echo "cert-file AND key-file";; + "2.2") + echo "true";; + "2.3") + echo "false";; + "2.4") + echo "peer-cert-file AND peer-key-file";; + "2.5") + echo "true";; + "2.6") + echo "--peer-auto-tls=false";; + "2.7") + echo "--trusted-ca-file";; + esac +fi + ``` -Verify that the `--service-account-key-file` argument exists and is set as appropriate. - -**Remediation:** -By default, K3s sets the `--service-account-key-file` explicitly. No manual remediation needed. - - -#### 1.2.29 -Ensure that the `--etcd-certfile` and `--etcd-keyfile` arguments are set as appropriate (Scored) -
-Rationale -etcd is a highly-available key-value store used by Kubernetes deployments for persistent storage of all of its REST API objects. These objects are sensitive in nature and should be protected by client authentication. This requires the API server to identify itself to the etcd server using a client certificate and key. -
- -**Result:** Pass - -**Audit:** -Run the below command on the master node. +**Audit Execution:** ```bash -journalctl -u k3s | grep "Running kube-apiserver" | tail -n1 | grep -E 'etcd-certfile|etcd-keyfile' +./check_for_k3s_etcd.sh 1.2.29 ``` -Verify that the `--etcd-certfile` and `--etcd-keyfile` arguments exist and they are set as appropriate. +**Expected Result**: + +```console +'--etcd-certfile' is present AND '--etcd-keyfile' is present +``` + +**Returned Value**: + +```console +Feb 21 23:13:24 k3s[5223]: time="2022-02-21T23:13:24.847339487Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit-log --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,NamespaceLifecycle,ServiceAccount --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --insecure-port=0 --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.30 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Automated) + + +**Result:** pass **Remediation:** -By default, K3s sets the `--etcd-certfile` and `--etcd-keyfile` arguments explicitly. No manual remediation needed. - - -#### 1.2.30 -Ensure that the `--tls-cert-file` and `--tls-private-key-file` arguments are set as appropriate (Scored) -
-Rationale -API server communication contains sensitive parameters that should remain encrypted in transit. Configure the API server to serve only HTTPS traffic. -
- -**Result:** Pass +Follow the Kubernetes documentation and set up the TLS connection on the apiserver. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the master node and set the TLS certificate and private key file parameters. +--tls-cert-file= +--tls-private-key-file= **Audit:** -Run the below command on the master node. ```bash -journalctl -u k3s | grep "Running kube-apiserver" | tail -n1 | grep -E 'tls-cert-file|tls-private-key-file' +journalctl -D /var/log/journal -u k3s | grep -A1 'Running kube-apiserver' | tail -n2 ``` -Verify that the `--tls-cert-file` and `--tls-private-key-file` arguments exist and they are set as appropriate. +**Expected Result**: + +```console +'--tls-cert-file' is present AND '--tls-private-key-file' is present +``` + +**Returned Value**: + +```console +Feb 21 23:13:24 k3s[5223]: time="2022-02-21T23:13:24.847339487Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit-log --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,NamespaceLifecycle,ServiceAccount --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --insecure-port=0 --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" Feb 21 23:13:24 k3s[5223]: {"level":"info","ts":"2022-02-21T23:13:24.848Z","caller":"raft/raft.go:1530","msg":"b3656202b34887ca switched to configuration voters=(12926846069174208458)"} +``` + +### 1.2.31 Ensure that the --client-ca-file argument is set as appropriate (Automated) + + +**Result:** pass **Remediation:** -By default, K3s sets the `--tls-cert-file` and `--tls-private-key-file` arguments explicitly. No manual remediation needed. - - -#### 1.2.31 -Ensure that the `--client-ca-file` argument is set as appropriate (Scored) -
-Rationale -API server communication contains sensitive parameters that should remain encrypted in transit. Configure the API server to serve only HTTPS traffic. If `--client-ca-file` argument is set, any request presenting a client certificate signed by one of the authorities in the `client-ca-file` is authenticated with an identity corresponding to the CommonName of the client certificate. -
- -**Result:** Pass +Follow the Kubernetes documentation and set up the TLS connection on the apiserver. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the master node and set the client certificate authority file. +--client-ca-file= **Audit:** -Run the below command on the master node. ```bash -journalctl -u k3s | grep "Running kube-apiserver" | tail -n1 | grep "client-ca-file" +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'client-ca-file' ``` -Verify that the `--client-ca-file` argument exists and it is set as appropriate. +**Expected Result**: + +```console +'--client-ca-file' is not present +``` + +### 1.2.32 Ensure that the --etcd-cafile argument is set as appropriate (Automated) + + +**Result:** pass **Remediation:** -By default, K3s sets the `--client-ca-file` argument explicitly. No manual remediation needed. - - -#### 1.2.32 -Ensure that the `--etcd-cafile` argument is set as appropriate (Scored) -
-Rationale -etcd is a highly-available key-value store used by Kubernetes deployments for persistent storage of all of its REST API objects. These objects are sensitive in nature and should be protected by client authentication. This requires the API server to identify itself to the etcd server using a SSL Certificate Authority file. -
- -**Result:** Pass +Follow the Kubernetes documentation and set up the TLS connection between the apiserver and etcd. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the master node and set the etcd certificate authority file parameter. +--etcd-cafile= **Audit:** -Run the below command on the master node. ```bash -journalctl -u k3s | grep "Running kube-apiserver" | tail -n1 | grep "etcd-cafile" +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-cafile' ``` -Verify that the `--etcd-cafile` argument exists and it is set as appropriate. +**Expected Result**: + +```console +'--etcd-cafile' is not present +``` + +### 1.2.33 Ensure that the --encryption-provider-config argument is set as appropriate (Manual) + + +**Result:** pass **Remediation:** -By default, K3s sets the `--etcd-cafile` argument explicitly. No manual remediation needed. - - -#### 1.2.33 -Ensure that the `--encryption-provider-config` argument is set as appropriate (Scored) -
-Rationale -etcd is a highly available key-value store used by Kubernetes deployments for persistent storage of all of its REST API objects. These objects are sensitive in nature and should be encrypted at rest to avoid any disclosures. -
- -**Result:** Pass +Follow the Kubernetes documentation and configure a EncryptionConfig file. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the master node and set the --encryption-provider-config parameter to the path of that file: --encryption-provider-config= **Audit:** -Run the below command on the master node. ```bash -journalctl -u k3s | grep "Running kube-apiserver" | tail -n1 | grep "encryption-provider-config" +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'encryption-provider-config' ``` -Verify that the `--encryption-provider-config` argument is set to a EncryptionConfigfile. Additionally, ensure that the `EncryptionConfigfile` has all the desired resources covered especially any secrets. +**Expected Result**: + +```console +'--encryption-provider-config' is not present +``` + +### 1.2.34 Ensure that encryption providers are appropriately configured (Manual) + + +**Result:** warn **Remediation:** -K3s server needs to be ran with the follow, `--kube-apiserver-arg='encryption-provider-config=/path/to/encryption_config'`. This can be done by running k3s with the `--secrets-encryptiuon` argument which will configure the encryption provider. - - -#### 1.2.34 -Ensure that encryption providers are appropriately configured (Scored) -
-Rationale -Where `etcd` encryption is used, it is important to ensure that the appropriate set of encryption providers is used. Currently, the `aescbc`, `kms` and `secretbox` are likely to be appropriate options. -
- -**Result:** Pass - -**Remediation:** -Follow the Kubernetes documentation and configure a `EncryptionConfig` file. -In this file, choose **aescbc**, **kms** or **secretbox** as the encryption provider. +Follow the Kubernetes documentation and configure a EncryptionConfig file. +In this file, choose aescbc, kms or secretbox as the encryption provider. **Audit:** -Run the below command on the master node. ```bash grep aescbc /path/to/encryption-config.json ``` -Run the below command on the master node. - -Verify that aescbc is set as the encryption provider for all the desired resources. - -**Remediation** -K3s server needs to be run with the following, `--secrets-encryption=true`, and verify that one of the allowed encryption providers is present. +### 1.2.35 Ensure that the API Server only makes use of Strong Cryptographic Ciphers (Manual) -#### 1.2.35 -Ensure that the API Server only makes use of Strong Cryptographic Ciphers (Not Scored) +**Result:** pass -
-Rationale -TLS ciphers have had a number of known vulnerabilities and weaknesses, which can reduce the protection provided by them. By default Kubernetes supports a number of TLS cipher suites including some that have security concerns, weakening the protection provided. -
- -**Result:** **Not Scored - Operator Dependent** +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the master node and set the below parameter. +--tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM +_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM +_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM +_SHA384 **Audit:** -Run the below command on the master node. ```bash -journalctl -u k3s | grep "Running kube-apiserver" | tail -n1 | grep "tls-cipher-suites" +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'tls-cipher-suites' ``` -Verify that the `--tls-cipher-suites` argument is set as outlined in the remediation procedure below. +**Expected Result**: + +```console +'--tls-cipher-suites' is not present +``` + +## 1.3 Controller Manager +### 1.3.1 Ensure that the --terminated-pod-gc-threshold argument is set as appropriate (Manual) + + +**Result:** pass **Remediation:** -By default, K3s explicitly doesn't set this flag. No manual remediation needed. - - -### 1.3 Controller Manager - -#### 1.3.1 -Ensure that the `--terminated-pod-gc-threshold` argument is set as appropriate (Not Scored) -
-Rationale -Garbage collection is important to ensure sufficient resource availability and avoiding degraded performance and availability. In the worst case, the system might crash or just be unusable for a long period of time. The current setting for garbage collection is 12,500 terminated pods which might be too high for your system to sustain. Based on your system resources and tests, choose an appropriate threshold value to activate garbage collection. -
- -**Result:** **Not Scored - Operator Dependent** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the master node and set the --terminated-pod-gc-threshold to an appropriate threshold, +for example: +--terminated-pod-gc-threshold=10 **Audit:** -Run the below command on the master node. ```bash -journalctl -u k3s | grep "Running kube-controller-manager" | tail -n1 | grep "terminated-pod-gc-threshold +journalctl -D /var/log/journal -u k3s | grep 'Running kube-controller-manager' | tail -n1 | grep 'terminated-pod-gc-threshold' ``` -Verify that the `--terminated-pod-gc-threshold` argument is set as appropriate. +**Expected Result**: + +```console +'--terminated-pod-gc-threshold' is not present +``` + +### 1.3.2 Ensure that the --profiling argument is set to false (Automated) + + +**Result:** pass **Remediation:** -K3s server needs to be run with the following, `--kube-controller-manager-arg='terminated-pod-gc-threshold=10`. - - -#### 1.3.2 -Ensure that the `--profiling` argument is set to false (Scored) -
-Rationale -Profiling allows for the identification of specific performance bottlenecks. It generates a significant amount of program data that could potentially be exploited to uncover system and program details. If you are not experiencing any bottlenecks and do not need the profiler for troubleshooting purposes, it is recommended to turn it off to reduce the potential attack surface. -
- -**Result:** Pass +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the master node and set the below parameter. +--profiling=false **Audit:** -Run the below command on the master node. ```bash -journalctl -u k3s | grep "Running kube-controller-manager" | tail -n1 | grep "profiling" +journalctl -D /var/log/journal -u k3s | grep 'Running kube-controller-manager' | tail -n1 | grep 'profiling' ``` -Verify that the `--profiling` argument is set to false. +**Expected Result**: + +```console +'--profiling' is not present +``` + +### 1.3.3 Ensure that the --use-service-account-credentials argument is set to true (Automated) + + +**Result:** pass **Remediation:** -By default, K3s sets the `--profiling` flag parameter to false. No manual remediation needed. - - -#### 1.3.3 -Ensure that the `--use-service-account-credentials` argument is set to `true` (Scored) -
-Rationale -The controller manager creates a service account per controller in the `kube-system` namespace, generates a credential for it, and builds a dedicated API client with that service account credential for each controller loop to use. Setting the `--use-service-account-credentials` to `true` runs each control loop within the controller manager using a separate service account credential. When used in combination with RBAC, this ensures that the control loops run with the minimum permissions required to perform their intended tasks. -
- -**Result:** Pass +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the master node to set the below parameter. +--use-service-account-credentials=true **Audit:** -Run the below command on the master node. ```bash -journalctl -u k3s | grep "Running kube-controller-manager" | tail -n1 | grep "use-service-account-credentials" +journalctl -D /var/log/journal -u k3s | grep 'Running kube-controller-manager' | tail -n1 | grep 'use-service-account-credentials' ``` -Verify that the `--use-service-account-credentials` argument is set to true. +**Expected Result**: + +```console +'--use-service-account-credentials' is not present +``` + +### 1.3.4 Ensure that the --service-account-private-key-file argument is set as appropriate (Automated) + + +**Result:** pass **Remediation:** -K3s server needs to be run with the following, `--kube-controller-manager-arg='use-service-account-credentials=true'` - - -#### 1.3.4 -Ensure that the `--service-account-private-key-file` argument is set as appropriate (Scored) -
-Rationale -To ensure that keys for service account tokens can be rotated as needed, a separate public/private key pair should be used for signing service account tokens. The private key should be specified to the controller manager with `--service-account-private-key-file` as appropriate. -
- -**Result:** Pass +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the master node and set the --service-account-private-key-file parameter +to the private key file for service accounts. +--service-account-private-key-file= **Audit:** -Run the below command on the master node. ```bash -journalctl -u k3s | grep "Running kube-controller-manager" | tail -n1 | grep "service-account-private-key-file" +journalctl -D /var/log/journal -u k3s | grep 'Running kube-controller-manager' | tail -n1 | grep 'service-account-private-key-file' ``` -Verify that the `--service-account-private-key-file` argument is set as appropriate. +**Expected Result**: + +```console +'--service-account-private-key-file' is not present +``` + +### 1.3.5 Ensure that the --root-ca-file argument is set as appropriate (Automated) + + +**Result:** pass **Remediation:** -By default, K3s sets the `--service-account-private-key-file` argument with the service account key file. No manual remediation needed. - - -#### 1.3.5 -Ensure that the `--root-ca-file` argument is set as appropriate (Scored) -
-Rationale -Processes running within pods that need to contact the API server must verify the API server's serving certificate. Failing to do so could be a subject to man-in-the-middle attacks. - -Providing the root certificate for the API server's serving certificate to the controller manager with the `--root-ca-file` argument allows the controller manager to inject the trusted bundle into pods so that they can verify TLS connections to the API server. -
- -**Result:** Pass +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the master node and set the --root-ca-file parameter to the certificate bundle file`. +--root-ca-file= **Audit:** -Run the below command on the master node. ```bash -journalctl -u k3s | grep "Running kube-controller-manager" | tail -n1 | grep "root-ca-file" +journalctl -D /var/log/journal -u k3s | grep 'Running kube-controller-manager' | tail -n1 | grep 'root-ca-file' ``` -Verify that the `--root-ca-file` argument exists and is set to a certificate bundle file containing the root certificate for the API server's serving certificate +**Expected Result**: -**Remediation:** -By default, K3s sets the `--root-ca-file` argument with the root ca file. No manual remediation needed. - - -#### 1.3.6 -Ensure that the `RotateKubeletServerCertificate` argument is set to `true` (Scored) -
-Rationale -`RotateKubeletServerCertificate` causes the kubelet to both request a serving certificate after bootstrapping its client credentials and rotate the certificate as its existing credentials expire. This automated periodic rotation ensures that there are no downtimes due to expired certificates and thus addressing availability in the CIA security triad. - -Note: This recommendation only applies if you let kubelets get their certificates from the API server. In case your kubelet certificates come from an outside authority/tool (e.g. Vault) then you need to take care of rotation yourself. -
- -**Result:** Not Applicable - -**Audit:** -Run the below command on the master node. - -```bash -journalctl -u k3s | grep "Running kube-controller-manager" | tail -n1 | grep "RotateKubeletServerCertificate" +```console +'--root-ca-file' is not present ``` -Verify that RotateKubeletServerCertificateargument exists and is set to true. +### 1.3.6 Ensure that the RotateKubeletServerCertificate argument is set to true (Automated) -**Remediation:** -By default, K3s implements its own logic for certificate generation and rotation. - - -#### 1.3.7 -Ensure that the `--bind-address` argument is set to `127.0.0.1` (Scored) -
-Rationale -The Controller Manager API service which runs on port 10252/TCP by default is used for health and metrics information and is available without authentication or encryption. As such it should only be bound to a localhost interface, to minimize the cluster's attack surface. -
- -**Result:** Pass - -**Audit:** -Run the below command on the master node. - -```bash -journalctl -u k3s | grep "Running kube-controller-manager" | tail -n1 | grep "bind-address" -``` - -Verify that the `--bind-address` argument is set to 127.0.0.1. - -**Remediation:** -By default, K3s sets the `--bind-address` argument to `127.0.0.1`. No manual remediation needed. - - -### 1.4 Scheduler -This section contains recommendations relating to Scheduler configuration flags - - -#### 1.4.1 -Ensure that the `--profiling` argument is set to `false` (Scored) -
-Rationale -Profiling allows for the identification of specific performance bottlenecks. It generates a significant amount of program data that could potentially be exploited to uncover system and program details. If you are not experiencing any bottlenecks and do not need the profiler for troubleshooting purposes, it is recommended to turn it off to reduce the potential attack surface. -
- -**Result:** Pass - -**Audit:** -Run the below command on the master node. - -```bash -journalctl -u k3s | grep "Running kube-scheduler" | tail -n1 | grep "profiling" -``` - -Verify that the `--profiling` argument is set to false. - -**Remediation:** -By default, K3s sets the `--profiling` flag parameter to false. No manual remediation needed. - - -#### 1.4.2 -Ensure that the `--bind-address` argument is set to `127.0.0.1` (Scored) -
-Rationale - -The Scheduler API service which runs on port 10251/TCP by default is used for health and metrics information and is available without authentication or encryption. As such it should only be bound to a localhost interface, to minimize the cluster's attack surface. -
- -**Result:** Pass - -**Audit:** -Run the below command on the master node. - -```bash -journalctl -u k3s | grep "Running kube-scheduler" | tail -n1 | grep "bind-address" -``` - -Verify that the `--bind-address` argument is set to 127.0.0.1. - -**Remediation:** -By default, K3s sets the `--bind-address` argument to `127.0.0.1`. No manual remediation needed. - - -## 2 Etcd Node Configuration -This section covers recommendations for etcd configuration. - -#### 2.1 -Ensure that the `cert-file` and `key-file` fields are set as appropriate (Scored) -
-Rationale -etcd is a highly-available key-value store used by Kubernetes deployments for persistent storage of all of its REST API objects. These objects are sensitive in nature and should be encrypted in transit. -
- -**Result:** Pass - -**Audit:** -Run the below command on the master node. - -```bash -grep -E 'cert-file|key-file' /var/lib/rancher/k3s/server/db/etcd/config -``` - -Verify that the `cert-file` and the `key-file` fields are set as appropriate. - -**Remediation:** -By default, K3s uses a config file for etcd that can be found at `/var/lib/rancher/k3s/server/db/etcd/config`. Server and peer cert and key files are specified. No manual remediation needed. - - -#### 2.2 -Ensure that the `client-cert-auth` field is set to `true` (Scored) -
-Rationale -etcd is a highly-available key-value store used by Kubernetes deployments for persistent storage of all of its REST API objects. These objects are sensitive in nature and should not be available to unauthenticated clients. You should enable the client authentication via valid certificates to secure the access to the etcd service. -
- -**Result:** Pass - -**Audit:** -Run the below command on the master node. - -```bash -grep 'client-cert-auth' /var/lib/rancher/k3s/server/db/etcd/config -``` - -Verify that the `client-cert-auth` field is set to true. - -**Remediation:** -By default, K3s uses a config file for etcd that can be found at `/var/lib/rancher/k3s/server/db/etcd/config`. `client-cert-auth` is set to true. No manual remediation needed. - - -#### 2.3 -Ensure that the `auto-tls` field is not set to `true` (Scored) -
-Rationale -etcd is a highly-available key-value store used by Kubernetes deployments for persistent storage of all of its REST API objects. These objects are sensitive in nature and should not be available to unauthenticated clients. You should enable the client authentication via valid certificates to secure the access to the etcd service. -
- -**Result:** Pass - -**Remediation:** -By default, K3s starts Etcd without this flag. It is set to `false` by default. - - -#### 2.4 -Ensure that the `peer-cert-file` and `peer-key-file` fields are set as appropriate (Scored) -
-Rationale -etcd is a highly-available key-value store used by Kubernetes deployments for persistent storage of all of its REST API objects. These objects are sensitive in nature and should be encrypted in transit and also amongst peers in the etcd clusters. -
- -**Result:** Pass - -**Remediation:** -By default, K3s starts Etcd with a config file found here, `/var/lib/rancher/k3s/server/db/etcd/config`. The config file contains `peer-transport-security:` which has fields that have the peer cert and peer key files. - - -#### 2.5 -Ensure that the `client-cert-auth` field is set to `true` (Scored) -
-Rationale -etcd is a highly-available key-value store used by Kubernetes deployments for persistent storage of all of its REST API objects. These objects are sensitive in nature and should be accessible only by authenticated etcd peers in the etcd cluster. -
- -**Result:** Pass - -**Audit:** -Run the below command on the master node. - -```bash -grep 'client-cert-auth' /var/lib/rancher/k3s/server/db/etcd/config -``` - -Verify that the `client-cert-auth` field in the peer section is set to true. - -**Remediation:** -By default, K3s uses a config file for etcd that can be found at `/var/lib/rancher/k3s/server/db/etcd/config`. Within the file, the `client-cert-auth` field is set. No manual remediation needed. - - -#### 2.6 -Ensure that the `peer-auto-tls` field is not set to `true` (Scored) -
-Rationale -etcd is a highly-available key-value store used by Kubernetes deployments for persistent storage of all of its REST API objects. These objects are sensitive in nature and should be accessible only by authenticated etcd peers in the etcd cluster. Hence, do not use self- signed certificates for authentication. -
- -**Result:** Pass - -**Audit:** -Run the below command on the master node. - -```bash -grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config -``` - -Verify that if the `peer-auto-tls` field does not exist. - -**Remediation:** -By default, K3s uses a config file for etcd that can be found at `/var/lib/rancher/k3s/server/db/etcd/config`. Within the file, it does not contain the `peer-auto-tls` field. No manual remediation needed. - - -#### 2.7 -Ensure that a unique Certificate Authority is used for etcd (Not Scored) -
-Rationale -etcd is a highly available key-value store used by Kubernetes deployments for persistent storage of all of its REST API objects. Its access should be restricted to specifically designated clients and peers only. - -Authentication to etcd is based on whether the certificate presented was issued by a trusted certificate authority. There is no checking of certificate attributes such as common name or subject alternative name. As such, if any attackers were able to gain access to any certificate issued by the trusted certificate authority, they would be able to gain full access to the etcd database. -
- -**Result:** Pass - -**Audit:** -Run the below command on the master node. - -```bash -# To find the ca file used by etcd: -grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config -# To find the kube-apiserver process: -journalctl -u k3s | grep "Running kube-apiserver" | tail -n1 -``` - -Verify that the file referenced by the `client-ca-file` flag in the apiserver process is different from the file referenced by the `trusted-ca-file` parameter in the etcd configuration file. - -**Remediation:** -By default, K3s uses a config file for etcd that can be found at `/var/lib/rancher/k3s/server/db/etcd/config` and the `trusted-ca-file` parameters in it are set to unique values specific to etcd. No manual remediation needed. - - - -## 3 Control Plane Configuration - - -### 3.1 Authentication and Authorization - - -#### 3.1.1 -Client certificate authentication should not be used for users (Not Scored) -
-Rationale -With any authentication mechanism the ability to revoke credentials if they are compromised or no longer required, is a key control. Kubernetes client certificate authentication does not allow for this due to a lack of support for certificate revocation. -
- -**Result:** Not Scored - Operator Dependent - -**Audit:** -Review user access to the cluster and ensure that users are not making use of Kubernetes client certificate authentication. - -**Remediation:** -Alternative mechanisms provided by Kubernetes such as the use of OIDC should be implemented in place of client certificates. - -### 3.2 Logging - - -#### 3.2.1 -Ensure that a minimal audit policy is created (Scored) -
-Rationale -Logging is an important detective control for all systems, to detect potential unauthorized access. -
- -**Result:** Does not pass. See the [Hardening Guide](../hardening_guide/) for details. - -**Audit:** -Run the below command on the master node. - -```bash -journalctl -u k3s | grep "Running kube-apiserver" | tail -n1 | grep "audit-policy-file" -``` - -Verify that the `--audit-policy-file` is set. Review the contents of the file specified and ensure that it contains a valid audit policy. - -**Remediation:** -Create an audit policy file for your cluster and pass it to k3s. e.g. `--kube-apiserver-arg='audit-log-path=/var/lib/rancher/k3s/server/logs/audit-log'` - - -#### 3.2.2 -Ensure that the audit policy covers key security concerns (Not Scored) -
-Rationale -Security audit logs should cover access and modification of key resources in the cluster, to enable them to form an effective part of a security environment. -
- -**Result:** Not Scored - Operator Dependent - -**Remediation:** - - -## 4 Worker Node Security Configuration - - -### 4.1 Worker Node Configuration Files - - -#### 4.1.1 -Ensure that the kubelet service file permissions are set to `644` or more restrictive (Scored) -
-Rationale -The `kubelet` service file controls various parameters that set the behavior of the kubelet service in the worker node. You should restrict its file permissions to maintain the integrity of the file. The file should be writable by only the administrators on the system. -
**Result:** Not Applicable **Remediation:** -K3s doesn’t launch the kubelet as a service. It is launched and managed by the K3s supervisor process. All configuration is passed to it as command line arguments at run time. +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the master node and set the --feature-gates parameter to include RotateKubeletServerCertificate=true. +--feature-gates=RotateKubeletServerCertificate=true + +### 1.3.7 Ensure that the --bind-address argument is set to 127.0.0.1 (Automated) -#### 4.1.2 -Ensure that the kubelet service file ownership is set to `root:root` (Scored) -
-Rationale -The `kubelet` service file controls various parameters that set the behavior of the kubelet service in the worker node. You should set its file ownership to maintain the integrity of the file. The file should be owned by `root:root`. -
+**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the master node and ensure the correct value for the --bind-address parameter + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-controller-manager' | tail -n1 | grep 'bind-address' +``` + +**Expected Result**: + +```console +'--bind-address' is present OR '--bind-address' is not present +``` + +## 1.4 Scheduler +### 1.4.1 Ensure that the --profiling argument is set to false (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Scheduler pod specification file /etc/kubernetes/manifests/kube-scheduler.yaml file +on the master node and set the below parameter. +--profiling=false + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-scheduler' | tail -n1 +``` + +**Expected Result**: + +```console +'false' is equal to 'false' +``` + +**Returned Value**: + +```console +Feb 21 23:13:24 k3s[5223]: time="2022-02-21T23:13:24.851975832Z" level=info msg="Running kube-scheduler --address=127.0.0.1 --bind-address=127.0.0.1 --kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --port=10251 --profiling=false --secure-port=0" +``` + +### 1.4.2 Ensure that the --bind-address argument is set to 127.0.0.1 (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Scheduler pod specification file /etc/kubernetes/manifests/kube-scheduler.yaml +on the master node and ensure the correct value for the --bind-address parameter + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-scheduler' | tail -n1 | grep 'bind-address' +``` + +**Expected Result**: + +```console +'--bind-address' is present OR '--bind-address' is not present +``` + +## 2 Etcd Node Configuration Files +### 2.1 Ensure that the --cert-file and --key-file arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the etcd service documentation and configure TLS encryption. +Then, edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml +on the master node and set the below parameters. +--cert-file= +--key-file= + +**Audit Script:** `check_for_k3s_etcd.sh` + +```bash +#!/bin/bash + +# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) +# before it checks the requirement +set -eE + +handle_error() { + echo "false" +} + +trap 'handle_error' ERR + + +if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd' | grep -v grep | wc -l)" -gt 0 ]]; then + case $1 in + "1.1.11") + echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; + "1.2.29") + echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; + "2.1") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.2") + echo "$(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth')";; + "2.3") + echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.4") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.5") + echo "$(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth')";; + "2.6") + echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.7") + echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; + esac +else +# If another database is running, return whatever is required to pass the scan + case $1 in + "1.1.11") + echo "700";; + "1.2.29") + echo "--etcd-certfile AND --etcd-keyfile";; + "2.1") + echo "cert-file AND key-file";; + "2.2") + echo "true";; + "2.3") + echo "false";; + "2.4") + echo "peer-cert-file AND peer-key-file";; + "2.5") + echo "true";; + "2.6") + echo "--peer-auto-tls=false";; + "2.7") + echo "--trusted-ca-file";; + esac +fi + +``` + +**Audit Execution:** + +```bash +./check_for_k3s_etcd.sh 2.1 +``` + +**Expected Result**: + +```console +'cert-file' is present AND 'key-file' is present +``` + +**Returned Value**: + +```console +cert-file: /var/lib/rancher/k3s/server/tls/etcd/server-client.crt key-file: /var/lib/rancher/k3s/server/tls/etcd/server-client.key +``` + +### 2.2 Ensure that the --client-cert-auth argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the etcd pod specification file /var/lib/rancher/k3s/server/db/etcd/config on the master +node and set the below parameter. +--client-cert-auth="true" + +**Audit Script:** `check_for_k3s_etcd.sh` + +```bash +#!/bin/bash + +# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) +# before it checks the requirement +set -eE + +handle_error() { + echo "false" +} + +trap 'handle_error' ERR + + +if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd' | grep -v grep | wc -l)" -gt 0 ]]; then + case $1 in + "1.1.11") + echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; + "1.2.29") + echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; + "2.1") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.2") + echo "$(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth')";; + "2.3") + echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.4") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.5") + echo "$(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth')";; + "2.6") + echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.7") + echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; + esac +else +# If another database is running, return whatever is required to pass the scan + case $1 in + "1.1.11") + echo "700";; + "1.2.29") + echo "--etcd-certfile AND --etcd-keyfile";; + "2.1") + echo "cert-file AND key-file";; + "2.2") + echo "true";; + "2.3") + echo "false";; + "2.4") + echo "peer-cert-file AND peer-key-file";; + "2.5") + echo "true";; + "2.6") + echo "--peer-auto-tls=false";; + "2.7") + echo "--trusted-ca-file";; + esac +fi + +``` + +**Audit Execution:** + +```bash +./check_for_k3s_etcd.sh 2.2 +``` + +**Expected Result**: + +```console +'--client-cert-auth' is not present +``` + +**Returned Value**: + +```console +client-cert-auth: true +``` + +### 2.3 Ensure that the --auto-tls argument is not set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the etcd pod specification file /var/lib/rancher/k3s/server/db/etcd/config on the master +node and either remove the --auto-tls parameter or set it to false. + --auto-tls=false + +**Audit Script:** `check_for_k3s_etcd.sh` + +```bash +#!/bin/bash + +# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) +# before it checks the requirement +set -eE + +handle_error() { + echo "false" +} + +trap 'handle_error' ERR + + +if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd' | grep -v grep | wc -l)" -gt 0 ]]; then + case $1 in + "1.1.11") + echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; + "1.2.29") + echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; + "2.1") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.2") + echo "$(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth')";; + "2.3") + echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.4") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.5") + echo "$(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth')";; + "2.6") + echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.7") + echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; + esac +else +# If another database is running, return whatever is required to pass the scan + case $1 in + "1.1.11") + echo "700";; + "1.2.29") + echo "--etcd-certfile AND --etcd-keyfile";; + "2.1") + echo "cert-file AND key-file";; + "2.2") + echo "true";; + "2.3") + echo "false";; + "2.4") + echo "peer-cert-file AND peer-key-file";; + "2.5") + echo "true";; + "2.6") + echo "--peer-auto-tls=false";; + "2.7") + echo "--trusted-ca-file";; + esac +fi + +``` + +**Audit Execution:** + +```bash +./check_for_k3s_etcd.sh 2.3 +``` + +**Expected Result**: + +```console +'--auto-tls' is not present OR '--auto-tls' is not present +``` + +**Returned Value**: + +```console +false +``` + +### 2.4 Ensure that the --peer-cert-file and --peer-key-file arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the etcd service documentation and configure peer TLS encryption as appropriate +for your etcd cluster. +Then, edit the etcd pod specification file /var/lib/rancher/k3s/server/db/etcd/config on the +master node and set the below parameters. +--peer-client-file= +--peer-key-file= + +**Audit Script:** `check_for_k3s_etcd.sh` + +```bash +#!/bin/bash + +# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) +# before it checks the requirement +set -eE + +handle_error() { + echo "false" +} + +trap 'handle_error' ERR + + +if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd' | grep -v grep | wc -l)" -gt 0 ]]; then + case $1 in + "1.1.11") + echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; + "1.2.29") + echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; + "2.1") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.2") + echo "$(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth')";; + "2.3") + echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.4") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.5") + echo "$(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth')";; + "2.6") + echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.7") + echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; + esac +else +# If another database is running, return whatever is required to pass the scan + case $1 in + "1.1.11") + echo "700";; + "1.2.29") + echo "--etcd-certfile AND --etcd-keyfile";; + "2.1") + echo "cert-file AND key-file";; + "2.2") + echo "true";; + "2.3") + echo "false";; + "2.4") + echo "peer-cert-file AND peer-key-file";; + "2.5") + echo "true";; + "2.6") + echo "--peer-auto-tls=false";; + "2.7") + echo "--trusted-ca-file";; + esac +fi + +``` + +**Audit Execution:** + +```bash +./check_for_k3s_etcd.sh 2.4 +``` + +**Expected Result**: + +```console +'cert-file' is present AND 'key-file' is present +``` + +**Returned Value**: + +```console +cert-file: /var/lib/rancher/k3s/server/tls/etcd/peer-server-client.crt key-file: /var/lib/rancher/k3s/server/tls/etcd/peer-server-client.key +``` + +### 2.5 Ensure that the --peer-client-cert-auth argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the etcd pod specification file /var/lib/rancher/k3s/server/db/etcd/config on the master +node and set the below parameter. +--peer-client-cert-auth=true + +**Audit Script:** `check_for_k3s_etcd.sh` + +```bash +#!/bin/bash + +# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) +# before it checks the requirement +set -eE + +handle_error() { + echo "false" +} + +trap 'handle_error' ERR + + +if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd' | grep -v grep | wc -l)" -gt 0 ]]; then + case $1 in + "1.1.11") + echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; + "1.2.29") + echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; + "2.1") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.2") + echo "$(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth')";; + "2.3") + echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.4") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.5") + echo "$(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth')";; + "2.6") + echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.7") + echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; + esac +else +# If another database is running, return whatever is required to pass the scan + case $1 in + "1.1.11") + echo "700";; + "1.2.29") + echo "--etcd-certfile AND --etcd-keyfile";; + "2.1") + echo "cert-file AND key-file";; + "2.2") + echo "true";; + "2.3") + echo "false";; + "2.4") + echo "peer-cert-file AND peer-key-file";; + "2.5") + echo "true";; + "2.6") + echo "--peer-auto-tls=false";; + "2.7") + echo "--trusted-ca-file";; + esac +fi + +``` + +**Audit Execution:** + +```bash +./check_for_k3s_etcd.sh 2.5 +``` + +**Expected Result**: + +```console +'--client-cert-auth' is not present +``` + +**Returned Value**: + +```console +client-cert-auth: true +``` + +### 2.6 Ensure that the --peer-auto-tls argument is not set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the etcd pod specification file /var/lib/rancher/k3s/server/db/etcd/config on the master +node and either remove the --peer-auto-tls parameter or set it to false. +--peer-auto-tls=false + +**Audit Script:** `check_for_k3s_etcd.sh` + +```bash +#!/bin/bash + +# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) +# before it checks the requirement +set -eE + +handle_error() { + echo "false" +} + +trap 'handle_error' ERR + + +if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd' | grep -v grep | wc -l)" -gt 0 ]]; then + case $1 in + "1.1.11") + echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; + "1.2.29") + echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; + "2.1") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.2") + echo "$(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth')";; + "2.3") + echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.4") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.5") + echo "$(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth')";; + "2.6") + echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.7") + echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; + esac +else +# If another database is running, return whatever is required to pass the scan + case $1 in + "1.1.11") + echo "700";; + "1.2.29") + echo "--etcd-certfile AND --etcd-keyfile";; + "2.1") + echo "cert-file AND key-file";; + "2.2") + echo "true";; + "2.3") + echo "false";; + "2.4") + echo "peer-cert-file AND peer-key-file";; + "2.5") + echo "true";; + "2.6") + echo "--peer-auto-tls=false";; + "2.7") + echo "--trusted-ca-file";; + esac +fi + +``` + +**Audit Execution:** + +```bash +./check_for_k3s_etcd.sh 2.6 +``` + +**Expected Result**: + +```console +'--peer-auto-tls' is not present OR '--peer-auto-tls' is present +``` + +**Returned Value**: + +```console +false +``` + +### 2.7 Ensure that a unique Certificate Authority is used for etcd (Manual) + + +**Result:** pass + +**Remediation:** +[Manual test] +Follow the etcd documentation and create a dedicated certificate authority setup for the +etcd service. +Then, edit the etcd pod specification file /var/lib/rancher/k3s/server/db/etcd/config on the +master node and set the below parameter. +--trusted-ca-file= + +**Audit Script:** `check_for_k3s_etcd.sh` + +```bash +#!/bin/bash + +# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) +# before it checks the requirement +set -eE + +handle_error() { + echo "false" +} + +trap 'handle_error' ERR + + +if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd' | grep -v grep | wc -l)" -gt 0 ]]; then + case $1 in + "1.1.11") + echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; + "1.2.29") + echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; + "2.1") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.2") + echo "$(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth')";; + "2.3") + echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.4") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.5") + echo "$(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth')";; + "2.6") + echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.7") + echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; + esac +else +# If another database is running, return whatever is required to pass the scan + case $1 in + "1.1.11") + echo "700";; + "1.2.29") + echo "--etcd-certfile AND --etcd-keyfile";; + "2.1") + echo "cert-file AND key-file";; + "2.2") + echo "true";; + "2.3") + echo "false";; + "2.4") + echo "peer-cert-file AND peer-key-file";; + "2.5") + echo "true";; + "2.6") + echo "--peer-auto-tls=false";; + "2.7") + echo "--trusted-ca-file";; + esac +fi + +``` + +**Audit Execution:** + +```bash +./check_for_k3s_etcd.sh 2.7 +``` + +**Expected Result**: + +```console +'trusted-ca-file' is present +``` + +**Returned Value**: + +```console +trusted-ca-file: /var/lib/rancher/k3s/server/tls/etcd/server-ca.crt trusted-ca-file: /var/lib/rancher/k3s/server/tls/etcd/peer-ca.crt +``` + +## 3.1 Authentication and Authorization +### 3.1.1 Client certificate authentication should not be used for users (Manual) + + +**Result:** warn + +**Remediation:** +Alternative mechanisms provided by Kubernetes such as the use of OIDC should be +implemented in place of client certificates. + +## 3.2 Logging +### 3.2.1 Ensure that a minimal audit policy is created (Manual) + + +**Result:** warn + +**Remediation:** +Create an audit policy file for your cluster. + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'audit-policy-file' +``` + +### 3.2.2 Ensure that the audit policy covers key security concerns (Manual) + + +**Result:** warn + +**Remediation:** +Consider modification of the audit policy in use on the cluster to include these items, at a +minimum. + +## 4.1 Worker Node Configuration Files +### 4.1.1 Ensure that the kubelet service file permissions are set to 644 or more restrictive (Automated) + **Result:** Not Applicable **Remediation:** -K3s doesn’t launch the kubelet as a service. It is launched and managed by the K3s supervisor process. All configuration is passed to it as command line arguments at run time. +Run the below command (based on the file location on your system) on the each worker node. +For example, +chmod 644 /etc/systemd/system/kubelet.service.d/10-kubeadm.conf +### 4.1.2 Ensure that the kubelet service file ownership is set to root:root (Automated) -#### 4.1.3 -Ensure that the proxy kubeconfig file permissions are set to `644` or more restrictive (Scored) -
-Rationale -The `kube-proxy` kubeconfig file controls various parameters of the `kube-proxy` service in the worker node. You should restrict its file permissions to maintain the integrity of the file. The file should be writable by only the administrators on the system. - -It is possible to run `kube-proxy` with the kubeconfig parameters configured as a Kubernetes ConfigMap instead of a file. In this case, there is no proxy kubeconfig file. -
**Result:** Not Applicable +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, +chown root:root /etc/systemd/system/kubelet.service.d/10-kubeadm.conf + +### 4.1.3 If proxy kubeconfig file exists ensure permissions are set to 644 or more restrictive (Manual) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, +chmod 644 /var/lib/rancher/k3s/agent/kubeproxy.kubeconfig + **Audit:** -Run the below command on the worker node. ```bash stat -c %a /var/lib/rancher/k3s/agent/kubeproxy.kubeconfig +``` + +**Expected Result**: + +```console +'permissions' is present OR '/var/lib/rancher/k3s/agent/kubeproxy.kubeconfig' is not present +``` + +**Returned Value**: + +```console 644 ``` -Verify that if a file is specified and it exists, the permissions are 644 or more restrictive. +### 4.1.4 Ensure that the proxy kubeconfig file ownership is set to root:root (Manual) + + +**Result:** pass **Remediation:** -K3s runs `kube-proxy` in process and does not use a config file. - - -#### 4.1.4 -Ensure that the proxy kubeconfig file ownership is set to `root:root` (Scored) -
-Rationale -The kubeconfig file for `kube-proxy` controls various parameters for the `kube-proxy` service in the worker node. You should set its file ownership to maintain the integrity of the file. The file should be owned by `root:root`. -
- -**Result:** Not Applicable +Run the below command (based on the file location on your system) on the each worker node. +For example, chown root:root /var/lib/rancher/k3s/agent/kubeproxy.kubeconfig **Audit:** -Run the below command on the master node. ```bash stat -c %U:%G /var/lib/rancher/k3s/agent/kubeproxy.kubeconfig +``` + +**Expected Result**: + +```console +'root:root' is not present OR '/var/lib/rancher/k3s/agent/kubeproxy.kubeconfig' is not present +``` + +**Returned Value**: + +```console root:root ``` -Verify that if a file is specified and it exists, the permissions are 644 or more restrictive. +### 4.1.5 Ensure that the --kubeconfig kubelet.conf file permissions are set to 644 or more restrictive (Automated) + + +**Result:** pass **Remediation:** -K3s runs `kube-proxy` in process and does not use a config file. - - -#### 4.1.5 -Ensure that the kubelet.conf file permissions are set to `644` or more restrictive (Scored) -
-Rationale -The `kubelet.conf` file is the kubeconfig file for the node, and controls various parameters that set the behavior and identity of the worker node. You should restrict its file permissions to maintain the integrity of the file. The file should be writable by only the administrators on the system. -
- -**Result:** Pass +Run the below command (based on the file location on your system) on the each worker node. +For example, +chmod 644 /var/lib/rancher/k3s/server/cred/admin.kubeconfig **Audit:** -Run the below command on the worker node. ```bash stat -c %a /var/lib/rancher/k3s/agent/kubelet.kubeconfig +``` + +**Expected Result**: + +```console +'644' is equal to '644' +``` + +**Returned Value**: + +```console 644 ``` +### 4.1.6 Ensure that the --kubeconfig kubelet.conf file ownership is set to root:root (Manual) + + +**Result:** warn + **Remediation:** -By default, K3s creates `kubelet.kubeconfig` with `644` permissions. No manual remediation needed. - -#### 4.1.6 -Ensure that the kubelet.conf file ownership is set to `root:root` (Scored) -
-Rationale -The `kubelet.conf` file is the kubeconfig file for the node, and controls various parameters that set the behavior and identity of the worker node. You should set its file ownership to maintain the integrity of the file. The file should be owned by `root:root`. -
- -**Result:** Not Applicable +Run the below command (based on the file location on your system) on the each worker node. +For example, +chown root:root /var/lib/rancher/k3s/server/cred/admin.kubeconfig **Audit:** -Run the below command on the master node. ```bash stat -c %U:%G /var/lib/rancher/k3s/agent/kubelet.kubeconfig -root:root ``` +### 4.1.7 Ensure that the certificate authorities file permissions are set to 644 or more restrictive (Manual) + + +**Result:** pass + **Remediation:** -By default, K3s creates `kubelet.kubeconfig` with `root:root` ownership. No manual remediation needed. - - -#### 4.1.7 -Ensure that the certificate authorities file permissions are set to `644` or more restrictive (Scored) -
-Rationale -The certificate authorities file controls the authorities used to validate API requests. You should restrict its file permissions to maintain the integrity of the file. The file should be writable by only the administrators on the system. -
- -**Result:** Pass +Run the following command to modify the file permissions of the +--client-ca-file chmod 644 **Audit:** -Run the below command on the master node. ```bash stat -c %a /var/lib/rancher/k3s/server/tls/server-ca.crt +``` + +**Expected Result**: + +```console +'644' is equal to '644' OR '640' is present OR '600' is present OR '444' is present OR '440' is present OR '400' is present OR '000' is present +``` + +**Returned Value**: + +```console 644 ``` -Verify that the permissions are 644. +### 4.1.8 Ensure that the client certificate authorities file ownership is set to root:root (Manual) + + +**Result:** warn **Remediation:** -By default, K3s creates `/var/lib/rancher/k3s/server/tls/server-ca.crt` with `644` permissions. - - -#### 4.1.8 -Ensure that the client certificate authorities file ownership is set to `root:root` (Scored) -
-Rationale -The certificate authorities file controls the authorities used to validate API requests. You should set its file ownership to maintain the integrity of the file. The file should be owned by `root:root`. -
- -**Result:** Pass +Run the following command to modify the ownership of the --client-ca-file. +chown root:root **Audit:** -Run the below command on the master node. ```bash stat -c %U:%G /var/lib/rancher/k3s/server/tls/client-ca.crt -root:root ``` -**Remediation:** -By default, K3s creates `/var/lib/rancher/k3s/server/tls/client-ca.crt` with `root:root` ownership. +### 4.1.9 Ensure that the kubelet --config configuration file has permissions set to 644 or more restrictive (Automated) -#### 4.1.9 -Ensure that the kubelet configuration file has permissions set to `644` or more restrictive (Scored) -
-Rationale -The kubelet reads various parameters, including security settings, from a config file specified by the `--config` argument. If this file is specified you should restrict its file permissions to maintain the integrity of the file. The file should be writable by only the administrators on the system. -
- **Result:** Not Applicable **Remediation:** -K3s doesn’t require or maintain a configuration file for the kubelet process. All configuration is passed to it as command line arguments at run time. +Run the following command (using the config file location identified in the Audit step) +chmod 644 /var/lib/kubelet/config.yaml +### 4.1.10 Ensure that the kubelet --config configuration file ownership is set to root:root (Automated) -#### 4.1.10 -Ensure that the kubelet configuration file ownership is set to `root:root` (Scored) -
-Rationale -The kubelet reads various parameters, including security settings, from a config file specified by the `--config` argument. If this file is specified you should restrict its file permissions to maintain the integrity of the file. The file should be owned by `root:root`. -
**Result:** Not Applicable **Remediation:** -K3s doesn’t require or maintain a configuration file for the kubelet process. All configuration is passed to it as command line arguments at run time. +Run the following command (using the config file location identified in the Audit step) +chown root:root /var/lib/kubelet/config.yaml + +## 4.2 Kubelet +### 4.2.1 Ensure that the anonymous-auth argument is set to false (Automated) -### 4.2 Kubelet -This section contains recommendations for kubelet configuration. - - -#### 4.2.1 -Ensure that the `--anonymous-auth` argument is set to false (Scored) -
-Rationale -When enabled, requests that are not rejected by other configured authentication methods are treated as anonymous requests. These requests are then served by the Kubelet server. You should rely on authentication to authorize access and disallow anonymous requests. -
- -**Result:** Pass - -**Audit:** -Run the below command on the master node. - -```bash -journalctl -u k3s | grep "Running kube-apiserver" | tail -n1 | grep "anonymous-auth" -``` - -Verify that the value for `--anonymous-auth` is false. +**Result:** pass **Remediation:** -By default, K3s starts kubelet with `--anonymous-auth` set to false. No manual remediation needed. - -#### 4.2.2 -Ensure that the `--authorization-mode` argument is not set to `AlwaysAllow` (Scored) -
-Rationale -Kubelets, by default, allow all authenticated requests (even anonymous ones) without needing explicit authorization checks from the apiserver. You should restrict this behavior and only allow explicitly authorized requests. -
- -**Result:** Pass +If using a Kubelet config file, edit the file to set authentication: anonymous: enabled to +false. +If using executable arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +--anonymous-auth=false +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service **Audit:** -Run the below command on the master node. ```bash -journalctl -u k3s | grep "Running kube-apiserver" | tail -n1 | grep "authorization-mode" +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'anonymous-auth' | grep -v grep ``` -Verify that `AlwaysAllow` is not present. +**Expected Result**: + +```console +'false' is equal to 'false' +``` + +**Returned Value**: + +```console +Feb 21 23:13:24 k3s[5223]: time="2022-02-21T23:13:24.847339487Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit-log --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,NamespaceLifecycle,ServiceAccount --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --insecure-port=0 --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 4.2.2 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated) + + +**Result:** pass **Remediation:** -K3s starts kubelet with `Webhook` as the value for the `--authorization-mode` argument. No manual remediation needed. - - -#### 4.2.3 -Ensure that the `--client-ca-file` argument is set as appropriate (Scored) -
-Rationale -The connections from the apiserver to the kubelet are used for fetching logs for pods, attaching (through kubectl) to running pods, and using the kubelet’s port-forwarding functionality. These connections terminate at the kubelet’s HTTPS endpoint. By default, the apiserver does not verify the kubelet’s serving certificate, which makes the connection subject to man-in-the-middle attacks, and unsafe to run over untrusted and/or public networks. Enabling Kubelet certificate authentication ensures that the apiserver could authenticate the Kubelet before submitting any requests. -
- -**Result:** Pass +If using a Kubelet config file, edit the file to set authorization: mode to Webhook. If +using executable arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_AUTHZ_ARGS variable. +--authorization-mode=Webhook +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service **Audit:** -Run the below command on the master node. ```bash -journalctl -u k3s | grep "Running kube-apiserver" | tail -n1 | grep "client-ca-file" +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'authorization-mode' | grep -v grep ``` -Verify that the `--client-ca-file` argument has a ca file associated. +**Expected Result**: + +```console +'Node,RBAC' not have 'AlwaysAllow' +``` + +**Returned Value**: + +```console +Feb 21 23:13:24 k3s[5223]: time="2022-02-21T23:13:24.847339487Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit-log --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,NamespaceLifecycle,ServiceAccount --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --insecure-port=0 --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 4.2.3 Ensure that the --client-ca-file argument is set as appropriate (Automated) + + +**Result:** pass **Remediation:** -By default, K3s starts the kubelet process with the `--client-ca-file`. No manual remediation needed. - - -#### 4.2.4 -Ensure that the `--read-only-port` argument is set to `0` (Scored) -
-Rationale -The Kubelet process provides a read-only API in addition to the main Kubelet API. Unauthenticated access is provided to this read-only API which could possibly retrieve potentially sensitive information about the cluster. -
- -**Result:** Pass +If using a Kubelet config file, edit the file to set authentication: x509: clientCAFile to +the location of the client CA file. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_AUTHZ_ARGS variable. +--client-ca-file= +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service **Audit:** -Run the below command on the master node. ```bash -journalctl -u k3s | grep "Running kubelet" | tail -n1 | grep "read-only-port" +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver'| tail -n1 | grep 'client-ca-file' | grep -v grep ``` -Verify that the `--read-only-port` argument is set to 0. + +**Expected Result**: + +```console +'--client-ca-file' is present +``` + +**Returned Value**: + +```console +Feb 21 23:13:24 k3s[5223]: time="2022-02-21T23:13:24.847339487Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit-log --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,NamespaceLifecycle,ServiceAccount --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --insecure-port=0 --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 4.2.4 Ensure that the --read-only-port argument is set to 0 (Manual) + + +**Result:** warn **Remediation:** -By default, K3s starts the kubelet process with the `--read-only-port` argument set to `0`. - - -#### 4.2.5 -Ensure that the `--streaming-connection-idle-timeout` argument is not set to `0` (Scored) -
-Rationale -Setting idle timeouts ensures that you are protected against Denial-of-Service attacks, inactive connections and running out of ephemeral ports. - -**Note:** By default, `--streaming-connection-idle-timeout` is set to 4 hours which might be too high for your environment. Setting this as appropriate would additionally ensure that such streaming connections are timed out after serving legitimate use cases. -
- -**Result:** Pass +If using a Kubelet config file, edit the file to set readOnlyPort to 0. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +--read-only-port=0 +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service **Audit:** -Run the below command on the master node. ```bash -journalctl -u k3s | grep "Running kubelet" | tail -n1 | grep "streaming-connection-idle-timeout" +journalctl -D /var/log/journal -u k3s | grep 'Running kubelet' | tail -n1 | grep 'read-only-port' ``` -Verify that there's nothing returned. +### 4.2.5 Ensure that the --streaming-connection-idle-timeout argument is not set to 0 (Manual) + + +**Result:** warn **Remediation:** -By default, K3s does not set `--streaming-connection-idle-timeout` when starting kubelet. - - -#### 4.2.6 -Ensure that the `--protect-kernel-defaults` argument is set to `true` (Scored) -
-Rationale -Kernel parameters are usually tuned and hardened by the system administrators before putting the systems into production. These parameters protect the kernel and the system. Your kubelet kernel defaults that rely on such parameters should be appropriately set to match the desired secured system state. Ignoring this could potentially lead to running pods with undesired kernel behavior. -
- -**Result:** Pass +If using a Kubelet config file, edit the file to set streamingConnectionIdleTimeout to a +value other than 0. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +--streaming-connection-idle-timeout=5m +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service **Audit:** -Run the below command on the master node. ```bash -journalctl -u k3s | grep "Running kubelet" | tail -n1 | grep "protect-kernel-defaults" +journalctl -D /var/log/journal -u k3s | grep 'Running kubelet' | tail -n1 | grep 'streaming-connection-idle-timeout' ``` +### 4.2.6 Ensure that the --protect-kernel-defaults argument is set to true (Automated) + + +**Result:** pass + **Remediation:** -K3s server needs to be started with the following, `--protect-kernel-defaults=true`. - - -#### 4.2.7 -Ensure that the `--make-iptables-util-chains` argument is set to `true` (Scored) -
-Rationale -Kubelets can automatically manage the required changes to iptables based on how you choose your networking options for the pods. It is recommended to let kubelets manage the changes to iptables. This ensures that the iptables configuration remains in sync with pods networking configuration. Manually configuring iptables with dynamic pod network configuration changes might hamper the communication between pods/containers and to the outside world. You might have iptables rules too restrictive or too open. -
- -**Result:** Pass +If using a Kubelet config file, edit the file to set protectKernelDefaults: true. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +--protect-kernel-defaults=true +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service **Audit:** -Run the below command on the master node. ```bash -journalctl -u k3s | grep "Running kubelet" | tail -n1 | grep "make-iptables-util-chains" +journalctl -D /var/log/journal -u k3s | grep 'Running kubelet' | tail -n1 | grep 'protect-kernel-defaults' ``` -Verify there are no results returned. +**Expected Result**: + +```console +'true' is equal to 'true' +``` + +**Returned Value**: + +```console +Feb 21 23:13:32 k3s[5223]: time="2022-02-21T23:13:32.581127632Z" level=info msg="Running kubelet --address=0.0.0.0 --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=cgroupfs --client-ca-file=/var/lib/rancher/k3s/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --cni-bin-dir=/var/lib/rancher/k3s/data/9de9bfcf367b723ef0ac73dd91761165a4a8ad11ad16a758d3a996264e60c612/bin --cni-conf-dir=/var/lib/rancher/k3s/agent/etc/cni/net.d --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --container-runtime=remote --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override= --kubeconfig=/var/lib/rancher/k3s/agent/kubelet.kubeconfig --make-iptables-util-chains=true --node-labels= --pod-manifest-path=/var/lib/rancher/k3s/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --streaming-connection-idle-timeout=5m --tls-cert-file=/var/lib/rancher/k3s/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/k3s/agent/serving-kubelet.key" +``` + +### 4.2.7 Ensure that the --make-iptables-util-chains argument is set to true (Automated) + + +**Result:** pass **Remediation:** -K3s server needs to be run with the following, `--kube-apiserver-arg='make-iptables-util-chains=true'`. +If using a Kubelet config file, edit the file to set makeIPTablesUtilChains: true. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +remove the --make-iptables-util-chains argument from the +KUBELET_SYSTEM_PODS_ARGS variable. +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kubelet' | tail -n1 | grep 'make-iptables-util-chains' +``` + +**Expected Result**: + +```console +'true' is equal to 'true' OR '--make-iptables-util-chains' is not present +``` + +**Returned Value**: + +```console +Feb 21 23:13:32 k3s[5223]: time="2022-02-21T23:13:32.581127632Z" level=info msg="Running kubelet --address=0.0.0.0 --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=cgroupfs --client-ca-file=/var/lib/rancher/k3s/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --cni-bin-dir=/var/lib/rancher/k3s/data/9de9bfcf367b723ef0ac73dd91761165a4a8ad11ad16a758d3a996264e60c612/bin --cni-conf-dir=/var/lib/rancher/k3s/agent/etc/cni/net.d --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --container-runtime=remote --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override= --kubeconfig=/var/lib/rancher/k3s/agent/kubelet.kubeconfig --make-iptables-util-chains=true --node-labels= --pod-manifest-path=/var/lib/rancher/k3s/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --streaming-connection-idle-timeout=5m --tls-cert-file=/var/lib/rancher/k3s/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/k3s/agent/serving-kubelet.key" +``` + +### 4.2.8 Ensure that the --hostname-override argument is not set (Manual) -#### 4.2.8 -Ensure that the `--hostname-override` argument is not set (Not Scored) -
-Rationale -Overriding hostnames could potentially break TLS setup between the kubelet and the apiserver. Additionally, with overridden hostnames, it becomes increasingly difficult to associate logs with a particular node and process them for security analytics. Hence, you should setup your kubelet nodes with resolvable FQDNs and avoid overriding the hostnames with IPs. -
**Result:** Not Applicable **Remediation:** -K3s does set this parameter for each host, but K3s also manages all certificates in the cluster. It ensures the hostname-override is included as a subject alternative name (SAN) in the kubelet's certificate. +Edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf +on each worker node and remove the --hostname-override argument from the +KUBELET_SYSTEM_PODS_ARGS variable. +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service + +### 4.2.9 Ensure that the --event-qps argument is set to 0 or a level which ensures appropriate event capture (Manual) -#### 4.2.9 -Ensure that the `--event-qps` argument is set to 0 or a level which ensures appropriate event capture (Not Scored) -
-Rationale -It is important to capture all events and not restrict event creation. Events are an important source of security information and analytics that ensure that your environment is consistently monitored using the event data. -
- -**Result:** Not Scored - Operator Dependent +**Result:** warn **Remediation:** -See CIS Benchmark guide for further details on configuring this. - -#### 4.2.10 -Ensure that the `--tls-cert-file` and `--tls-private-key-file` arguments are set as appropriate (Scored) -
-Rationale -Kubelet communication contains sensitive parameters that should remain encrypted in transit. Configure the Kubelets to serve only HTTPS traffic. -
- -**Result:** Pass +If using a Kubelet config file, edit the file to set eventRecordQPS: to an appropriate level. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service **Audit:** -Run the below command on the master node. ```bash -journalctl -u k3s | grep "Running kubelet" | tail -n1 | grep -E 'tls-cert-file|tls-private-key-file' +/bin/ps -fC containerd ``` -Verify the `--tls-cert-file` and `--tls-private-key-file` arguments are present and set appropriately. +### 4.2.10 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Manual) + + +**Result:** warn **Remediation:** -By default, K3s sets the `--tls-cert-file` and `--tls-private-key-file` arguments when executing the kubelet process. +If using a Kubelet config file, edit the file to set tlsCertFile to the location +of the certificate file to use to identify this Kubelet, and tlsPrivateKeyFile +to the location of the corresponding private key file. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameters in KUBELET_CERTIFICATE_ARGS variable. +--tls-cert-file= +--tls-private-key-file= +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service +**Audit:** -#### 4.2.11 -Ensure that the `--rotate-certificates` argument is not set to `false` (Scored) -
-Rationale +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kubelet' | tail -n1 +``` -The `--rotate-certificates` setting causes the kubelet to rotate its client certificates by creating new CSRs as its existing credentials expire. This automated periodic rotation ensures that there is no downtime due to expired certificates and thus addressing availability in the CIA security triad. +### 4.2.11 Ensure that the --rotate-certificates argument is not set to false (Manual) -**Note:** This recommendation only applies if you let kubelets get their certificates from the API server. In case your kubelet certificates come from an outside authority/tool (e.g. Vault) then you need to take care of rotation yourself. - -**Note:**This feature also requires the `RotateKubeletClientCertificate` feature gate to be enabled (which is the default since Kubernetes v1.7) -
**Result:** Not Applicable **Remediation:** -By default, K3s implements its own logic for certificate generation and rotation. +If using a Kubelet config file, edit the file to add the line rotateCertificates: true or +remove it altogether to use the default value. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +remove --rotate-certificates=false argument from the KUBELET_CERTIFICATE_ARGS +variable. +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service +### 4.2.12 Verify that the RotateKubeletServerCertificate argument is set to true (Manual) -#### 4.2.12 -Ensure that the `RotateKubeletServerCertificate` argument is set to `true` (Scored) -
-Rationale -`RotateKubeletServerCertificate` causes the kubelet to both request a serving certificate after bootstrapping its client credentials and rotate the certificate as its existing credentials expire. This automated periodic rotation ensures that there are no downtimes due to expired certificates and thus addressing availability in the CIA security triad. - -Note: This recommendation only applies if you let kubelets get their certificates from the API server. In case your kubelet certificates come from an outside authority/tool (e.g. Vault) then you need to take care of rotation yourself. -
**Result:** Not Applicable **Remediation:** -By default, K3s implements its own logic for certificate generation and rotation. +Edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf +on each worker node and set the below parameter in KUBELET_CERTIFICATE_ARGS variable. +--feature-gates=RotateKubeletServerCertificate=true +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service + +### 4.2.13 Ensure that the Kubelet only makes use of Strong Cryptographic Ciphers (Manual) -#### 4.2.13 -Ensure that the Kubelet only makes use of Strong Cryptographic Ciphers (Not Scored) -
-Rationale -TLS ciphers have had a number of known vulnerabilities and weaknesses, which can reduce the protection provided by them. By default Kubernetes supports a number of TLS ciphersuites including some that have security concerns, weakening the protection provided. -
- -**Result:** Not Scored - Operator Dependent +**Result:** warn **Remediation:** -Configuration of the parameter is dependent on your use case. Please see the CIS Kubernetes Benchmark for suggestions on configuring this for your use-case. - - -## 5 Kubernetes Policies - - -### 5.1 RBAC and Service Accounts - - -#### 5.1.1 -Ensure that the cluster-admin role is only used where required (Not Scored) -
-Rationale -Kubernetes provides a set of default roles where RBAC is used. Some of these roles such as `cluster-admin` provide wide-ranging privileges which should only be applied where absolutely necessary. Roles such as `cluster-admin` allow super-user access to perform any action on any resource. When used in a `ClusterRoleBinding`, it gives full control over every resource in the cluster and in all namespaces. When used in a `RoleBinding`, it gives full control over every resource in the rolebinding's namespace, including the namespace itself. -
- -**Result:** Pass - -**Remediation:** -K3s does not make inappropriate use of the cluster-admin role. Operators must audit their workloads of additional usage. See the CIS Benchmark guide for more details. - -#### 5.1.2 -Minimize access to secrets (Not Scored) -
-Rationale -Inappropriate access to secrets stored within the Kubernetes cluster can allow for an attacker to gain additional access to the Kubernetes cluster or external resources whose credentials are stored as secrets. -
- -**Result:** Not Scored - Operator Dependent - -**Remediation:** -K3s limits its use of secrets for the system components appropriately, but operators must audit the use of secrets by their workloads. See the CIS Benchmark guide for more details. - -#### 5.1.3 -Minimize wildcard use in Roles and ClusterRoles (Not Scored) -
-Rationale -The principle of least privilege recommends that users are provided only the access required for their role and nothing more. The use of wildcard rights grants is likely to provide excessive rights to the Kubernetes API. -
- -**Result:** Not Scored - Operator Dependent +If using a Kubelet config file, edit the file to set TLSCipherSuites: to +TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 +or to a subset of these values. +If using executable arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the --tls-cipher-suites parameter as follows, or to a subset of these values. +--tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service **Audit:** -Run the below command on the master node. ```bash -# Retrieve the roles defined across each namespaces in the cluster and review for wildcards -kubectl get roles --all-namespaces -o yaml - -# Retrieve the cluster roles defined in the cluster and review for wildcards -kubectl get clusterroles -o yaml +/bin/ps -fC containerd ``` -Verify that there are not wildcards in use. +## 5.1 RBAC and Service Accounts +### 5.1.1 Ensure that the cluster-admin role is only used where required (Manual) + + +**Result:** warn **Remediation:** -Operators should review their workloads for proper role usage. See the CIS Benchmark guide for more details. +Identify all clusterrolebindings to the cluster-admin role. Check if they are used and +if they need this role or if they could use a role with fewer privileges. +Where possible, first bind users to a lower privileged role and then remove the +clusterrolebinding to the cluster-admin role : +kubectl delete clusterrolebinding [name] -#### 5.1.4 -Minimize access to create pods (Not Scored) -
-Rationale -The ability to create pods in a cluster opens up possibilities for privilege escalation and should be restricted, where possible. -
+### 5.1.2 Minimize access to secrets (Manual) -**Result:** Not Scored - Operator Dependent + +**Result:** warn **Remediation:** -Operators should review who has access to create pods in their cluster. See the CIS Benchmark guide for more details. +Where possible, remove get, list and watch access to secret objects in the cluster. -#### 5.1.5 -Ensure that default service accounts are not actively used. (Scored) -
-Rationale -Kubernetes provides a default service account which is used by cluster workloads where no specific service account is assigned to the pod. +### 5.1.3 Minimize wildcard use in Roles and ClusterRoles (Manual) -Where access to the Kubernetes API from a pod is required, a specific service account should be created for that pod, and rights granted to that service account. -The default service account should be configured such that it does not provide a service account token and does not have any explicit rights assignments. -
+**Result:** warn -**Result:** Fail. Currently requires operator intervention See the [Hardening Guide]({{}}/k3s/latest/en/security/hardening_guide) for details. +**Remediation:** +Where possible replace any use of wildcards in clusterroles and roles with specific +objects or actions. -**Audit:** -For each namespace in the cluster, review the rights assigned to the default service account and ensure that it has no roles or cluster roles bound to it apart from the defaults. Additionally ensure that the automountServiceAccountToken: false setting is in place for each default service account. +### 5.1.4 Minimize access to create pods (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove create access to pod objects in the cluster. + +### 5.1.5 Ensure that default service accounts are not actively used. (Manual) + + +**Result:** warn **Remediation:** Create explicit service accounts wherever a Kubernetes workload requires specific access to the Kubernetes API server. Modify the configuration of each default service account to include this value - -``` bash automountServiceAccountToken: false -``` + +### 5.1.6 Ensure that Service Account Tokens are only mounted where necessary (Manual) -#### 5.1.6 -Ensure that Service Account Tokens are only mounted where necessary (Not Scored) -
-Rationale -Mounting service account tokens inside pods can provide an avenue for privilege escalation attacks where an attacker is able to compromise a single pod in the cluster. - -Avoiding mounting these tokens removes this attack avenue. -
- -**Result:** Not Scored - Operator Dependent +**Result:** warn **Remediation:** -The pods launched by K3s are part of the control plane and generally need access to communicate with the API server, thus this control does not apply to them. Operators should review their workloads and take steps to modify the definition of pods and service accounts which do not need to mount service account tokens to disable it. +Modify the definition of pods and service accounts which do not need to mount service +account tokens to disable it. -### 5.2 Pod Security Policies +## 5.2 Pod Security Policies +### 5.2.1 Minimize the admission of privileged containers (Manual) -#### 5.2.1 -Minimize the admission of containers wishing to share the host process ID namespace (Scored) -
-Rationale -Privileged containers have access to all Linux Kernel capabilities and devices. A container running with full privileges can do almost everything that the host can do. This flag exists to allow special use-cases, like manipulating the network stack and accessing devices. +**Result:** warn -There should be at least one PodSecurityPolicy (PSP) defined which does not permit privileged containers. - -If you need to run privileged containers, this should be defined in a separate PSP and you should carefully check RBAC controls to ensure that only limited service accounts and users are given permission to access that PSP. -
- -**Result:** Pass +**Remediation:** +Create a PSP as described in the Kubernetes documentation, ensuring that +the .spec.privileged field is omitted or set to false. **Audit:** -Run the below command on the master node. ```bash -kubectl describe psp | grep MustRunAsNonRoot +kubectl describe psp global-restricted-psp | grep MustRunAsNonRoot ``` -Verify that the result is `Rule: MustRunAsNonRoot`. +### 5.2.2 Minimize the admission of containers wishing to share the host process ID namespace (Manual) + + +**Result:** pass **Remediation:** -An operator should apply a PodSecurityPolicy that sets the `Rule` value to `MustRunAsNonRoot`. An example of this can be found in the [Hardening Guide](../hardening_guide/). - - -#### 5.2.2 -Minimize the admission of containers wishing to share the host process ID namespace (Scored) -
-Rationale -A container running in the host's PID namespace can inspect processes running outside the container. If the container also has access to ptrace capabilities this can be used to escalate privileges outside of the container. - -There should be at least one PodSecurityPolicy (PSP) defined which does not permit containers to share the host PID namespace. - -If you need to run containers which require hostPID, this should be defined in a separate PSP and you should carefully check RBAC controls to ensure that only limited service accounts and users are given permission to access that PSP. -
- -**Result:** Pass +Create a PSP as described in the Kubernetes documentation, ensuring that the +.spec.hostPID field is omitted or set to false. **Audit:** -Run the below command on the master node. ```bash kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.hostPID == null) or (.spec.hostPID == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' ``` -Verify that the returned count is 1. +**Expected Result**: + +```console +1 is greater than 0 +``` + +**Returned Value**: + +```console +--count=1 +``` + +### 5.2.3 Minimize the admission of containers wishing to share the host IPC namespace (Manual) + + +**Result:** pass **Remediation:** -An operator should apply a PodSecurityPolicy that sets the `hostPID` value to false explicitly for the PSP it creates. An example of this can be found in the [Hardening Guide](../hardening_guide/). - - -#### 5.2.3 -Minimize the admission of containers wishing to share the host IPC namespace (Scored) -
-Rationale - -A container running in the host's IPC namespace can use IPC to interact with processes outside the container. - -There should be at least one PodSecurityPolicy (PSP) defined which does not permit containers to share the host IPC namespace. - -If you have a requirement to containers which require hostIPC, this should be defined in a separate PSP and you should carefully check RBAC controls to ensure that only limited service accounts and users are given permission to access that PSP. -
- -**Result:** Pass +Create a PSP as described in the Kubernetes documentation, ensuring that the +.spec.hostIPC field is omitted or set to false. **Audit:** -Run the below command on the master node. ```bash kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.hostIPC == null) or (.spec.hostIPC == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' ``` -Verify that the returned count is 1. +**Expected Result**: + +```console +1 is greater than 0 +``` + +**Returned Value**: + +```console +--count=1 +``` + +### 5.2.4 Minimize the admission of containers wishing to share the host network namespace (Manual) + + +**Result:** pass **Remediation:** -An operator should apply a PodSecurityPolicy that sets the `HostIPC` value to false explicitly for the PSP it creates. An example of this can be found in the [Hardening Guide](../hardening_guide/). - - -#### 5.2.4 -Minimize the admission of containers wishing to share the host network namespace (Scored) -
-Rationale -A container running in the host's network namespace could access the local loopback device, and could access network traffic to and from other pods. - -There should be at least one PodSecurityPolicy (PSP) defined which does not permit containers to share the host network namespace. - -If you have need to run containers which require hostNetwork, this should be defined in a separate PSP and you should carefully check RBAC controls to ensure that only limited service accounts and users are given permission to access that PSP. -
- -**Result:** Pass +Create a PSP as described in the Kubernetes documentation, ensuring that the +.spec.hostNetwork field is omitted or set to false. **Audit:** -Run the below command on the master node. ```bash kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.hostNetwork == null) or (.spec.hostNetwork == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' ``` -Verify that the returned count is 1. +**Expected Result**: + +```console +1 is greater than 0 +``` + +**Returned Value**: + +```console +--count=1 +``` + +### 5.2.5 Minimize the admission of containers with allowPrivilegeEscalation (Manual) + + +**Result:** pass **Remediation:** -An operator should apply a PodSecurityPolicy that sets the `HostNetwork` value to false explicitly for the PSP it creates. An example of this can be found in the [Hardening Guide](../hardening_guide/). - - -#### 5.2.5 -Minimize the admission of containers with `allowPrivilegeEscalation` (Scored) -
-Rationale -A container running with the `allowPrivilegeEscalation` flag set to true may have processes that can gain more privileges than their parent. - -There should be at least one PodSecurityPolicy (PSP) defined which does not permit containers to allow privilege escalation. The option exists (and is defaulted to true) to permit setuid binaries to run. - -If you have need to run containers which use setuid binaries or require privilege escalation, this should be defined in a separate PSP and you should carefully check RBAC controls to ensure that only limited service accounts and users are given permission to access that PSP. -
- -**Result:** Pass +Create a PSP as described in the Kubernetes documentation, ensuring that the +.spec.allowPrivilegeEscalation field is omitted or set to false. **Audit:** -Run the below command on the master node. ```bash kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.allowPrivilegeEscalation == null) or (.spec.allowPrivilegeEscalation == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' ``` -Verify that the returned count is 1. +**Expected Result**: + +```console +1 is greater than 0 +``` + +**Returned Value**: + +```console +--count=1 +``` + +### 5.2.6 Minimize the admission of root containers (Manual) + + +**Result:** pass **Remediation:** -An operator should apply a PodSecurityPolicy that sets the `allowPrivilegeEscalation` value to false explicitly for the PSP it creates. An example of this can be found in the [Hardening Guide](../hardening_guide/). - - -#### 5.2.6 -Minimize the admission of root containers (Not Scored) -
-Rationale -Containers may run as any Linux user. Containers which run as the root user, whilst constrained by Container Runtime security features still have an escalated likelihood of container breakout. - -Ideally, all containers should run as a defined non-UID 0 user. - -There should be at least one PodSecurityPolicy (PSP) defined which does not permit root users in a container. - -If you need to run root containers, this should be defined in a separate PSP and you should carefully check RBAC controls to ensure that only limited service accounts and users are given permission to access that PSP. -
- -**Result:** Not Scored +Create a PSP as described in the Kubernetes documentation, ensuring that the +.spec.runAsUser.rule is set to either MustRunAsNonRoot or MustRunAs with the range of +UIDs not including 0. **Audit:** -Run the below command on the master node. ```bash kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.allowPrivilegeEscalation == null) or (.spec.allowPrivilegeEscalation == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' ``` -Verify that the returned count is 1. +**Expected Result**: -**Remediation:** -An operator should apply a PodSecurityPolicy that sets the `runAsUser.Rule` value to `MustRunAsNonRoot`. An example of this can be found in the [Hardening Guide](../hardening_guide/). - - -#### 5.2.7 -Minimize the admission of containers with the NET_RAW capability (Not Scored) -
-Rationale -Containers run with a default set of capabilities as assigned by the Container Runtime. By default this can include potentially dangerous capabilities. With Docker as the container runtime the NET_RAW capability is enabled which may be misused by malicious containers. - -Ideally, all containers should drop this capability. - -There should be at least one PodSecurityPolicy (PSP) defined which prevents containers with the NET_RAW capability from launching. - -If you need to run containers with this capability, this should be defined in a separate PSP and you should carefully check RBAC controls to ensure that only limited service accounts and users are given permission to access that PSP. -
- -**Result:** Not Scored - -**Audit:** -Run the below command on the master node. - -```bash -kubectl get psp -o json | jq .spec.requiredDropCapabilities[] +```console +1 is greater than 0 ``` -Verify the value is `"ALL"`. +**Returned Value**: + +```console +--count=1 +``` + +### 5.2.7 Minimize the admission of containers with the NET_RAW capability (Manual) + + +**Result:** warn **Remediation:** -An operator should apply a PodSecurityPolicy that sets `.spec.requiredDropCapabilities[]` to a value of `All`. An example of this can be found in the [Hardening Guide](../hardening_guide/). - - -#### 5.2.8 -Minimize the admission of containers with added capabilities (Not Scored) -
-Rationale -Containers run with a default set of capabilities as assigned by the Container Runtime. Capabilities outside this set can be added to containers which could expose them to risks of container breakout attacks. - -There should be at least one PodSecurityPolicy (PSP) defined which prevents containers with capabilities beyond the default set from launching. - -If you need to run containers with additional capabilities, this should be defined in a separate PSP and you should carefully check RBAC controls to ensure that only limited service accounts and users are given permission to access that PSP. -
- -**Result:** Not Scored +Create a PSP as described in the Kubernetes documentation, ensuring that the +.spec.requiredDropCapabilities is set to include either NET_RAW or ALL. **Audit:** -Run the below command on the master node. ```bash kubectl get psp ``` -Verify that there are no PSPs present which have `allowedCapabilities` set to anything other than an empty array. +### 5.2.8 Minimize the admission of containers with added capabilities (Manual) + + +**Result:** warn **Remediation:** -An operator should apply a PodSecurityPolicy that sets `allowedCapabilities` to anything other than an empty array. An example of this can be found in the [Hardening Guide](../hardening_guide/). +Ensure that allowedCapabilities is not present in PSPs for the cluster unless +it is set to an empty array. + +### 5.2.9 Minimize the admission of containers with capabilities assigned (Manual) -#### 5.2.9 -Minimize the admission of containers with capabilities assigned (Not Scored) -
-Rationale -Containers run with a default set of capabilities as assigned by the Container Runtime. Capabilities are parts of the rights generally granted on a Linux system to the root user. +**Result:** warn -In many cases applications running in containers do not require any capabilities to operate, so from the perspective of the principle of least privilege use of capabilities should be minimized. -
+**Remediation:** +Review the use of capabilites in applications runnning on your cluster. Where a namespace +contains applicaions which do not require any Linux capabities to operate consider adding +a PSP which forbids the admission of containers which do not drop all capabilities. -**Result:** Not Scored +## 5.3 Network Policies and CNI +### 5.3.1 Ensure that the CNI in use supports Network Policies (Manual) -**Audit:** -Run the below command on the master node. + +**Result:** warn + +**Remediation:** +If the CNI plugin in use does not support network policies, consideration should be given to +making use of a different plugin, or finding an alternate mechanism for restricting traffic +in the Kubernetes cluster. + +### 5.3.2 Ensure that all Namespaces have Network Policies defined (Manual) + + +**Result:** pass + +**Remediation:** +Follow the documentation and create NetworkPolicy objects as you need them. + +**Audit Script:** `check_for_rke2_network_policies.sh` ```bash -kubectl get psp -``` +#!/bin/bash -**Remediation:** -An operator should apply a PodSecurityPolicy that sets `requiredDropCapabilities` to `ALL`. An example of this can be found in the [Hardening Guide](../hardening_guide/). +set -eE +handle_error() { + echo "false" +} -### 5.3 Network Policies and CNI +trap 'handle_error' ERR - -#### 5.3.1 -Ensure that the CNI in use supports Network Policies (Not Scored) -
-Rationale -Kubernetes network policies are enforced by the CNI plugin in use. As such it is important to ensure that the CNI plugin supports both Ingress and Egress network policies. -
- -**Result:** Pass - -**Audit:** -Review the documentation of CNI plugin in use by the cluster, and confirm that it supports Ingress and Egress network policies. - -**Remediation:** -By default, K3s use Canal (Calico and Flannel) and fully supports network policies. - - -#### 5.3.2 -Ensure that all Namespaces have Network Policies defined (Scored) -
-Rationale -Running different applications on the same Kubernetes cluster creates a risk of one compromised application attacking a neighboring application. Network segmentation is important to ensure that containers can communicate only with those they are supposed to. A network policy is a specification of how selections of pods are allowed to communicate with each other and other network endpoints. - -Network Policies are namespace scoped. When a network policy is introduced to a given namespace, all traffic not allowed by the policy is denied. However, if there are no network policies in a namespace all traffic will be allowed into and out of the pods in that namespace. -
- -**Result:** Pass - -**Audit:** -Run the below command on the master node. - -```bash -for i in kube-system kube-public default; do - kubectl get networkpolicies -n $i; +for namespace in kube-system kube-public default; do + policy_count=$(/var/lib/rancher/rke2/bin/kubectl get networkpolicy -n ${namespace} -o json | jq -r '.items | length') + if [ ${policy_count} -eq 0 ]; then + echo "false" + exit + fi done + +echo "true" + ``` -Verify that there are network policies applied to each of the namespaces. - -**Remediation:** -An operator should apply NetworkPolcyies that prevent unneeded traffic from traversing networks unnecessarily. An example of applying a NetworkPolcy can be found in the [Hardening Guide](../hardening_guide/). - -### 5.4 Secrets Management - - -#### 5.4.1 -Prefer using secrets as files over secrets as environment variables (Not Scored) -
-Rationale -It is reasonably common for application code to log out its environment (particularly in the event of an error). This will include any secret values passed in as environment variables, so secrets can easily be exposed to any user or entity who has access to the logs. -
- -**Result:** Not Scored - -**Audit:** -Run the following command to find references to objects which use environment variables defined from secrets. +**Audit Execution:** ```bash -kubectl get all -o jsonpath='{range .items[?(@..secretKeyRef)]} {.kind} {.metadata.name} {"\n"}{end}' -A +./check_for_rke2_network_policies.sh ``` -**Remediation:** -If possible, rewrite application code to read secrets from mounted secret files, rather than from environment variables. +**Expected Result**: + +```console +'true' is equal to 'true' +``` + +**Returned Value**: + +```console +true +``` + +## 5.4 Secrets Management +### 5.4.1 Prefer using secrets as files over secrets as environment variables (Manual) -#### 5.4.2 -Consider external secret storage (Not Scored) -
-Rationale -Kubernetes supports secrets as first-class objects, but care needs to be taken to ensure that access to secrets is carefully limited. Using an external secrets provider can ease the management of access to secrets, especially where secrets are used across both Kubernetes and non-Kubernetes environments. -
- -**Result:** Not Scored - -**Audit:** -Review your secrets management implementation. +**Result:** warn **Remediation:** -Refer to the secrets management options offered by your cloud provider or a third-party secrets management solution. - - -### 5.5 Extensible Admission Control - - -#### 5.5.1 -Configure Image Provenance using ImagePolicyWebhook admission controller (Not Scored) -
-Rationale -Kubernetes supports plugging in provenance rules to accept or reject the images in your deployments. You could configure such rules to ensure that only approved images are deployed in the cluster. -
- -**Result:** Not Scored +if possible, rewrite application code to read secrets from mounted secret files, rather than +from environment variables. **Audit:** -Review the pod definitions in your cluster and verify that image _provenance_ is configured as appropriate. + +```bash +kubectl get all -o jsonpath='{range .items[?(@..secretKeyRef)]} {.kind} {.metadata.name} {' '}{end}' -A +``` + +### 5.4.2 Consider external secret storage (Manual) + + +**Result:** warn + +**Remediation:** +Refer to the secrets management options offered by your cloud provider or a third-party +secrets management solution. + +## 5.5 Extensible Admission Control +### 5.5.1 Configure Image Provenance using ImagePolicyWebhook admission controller (Manual) + + +**Result:** warn **Remediation:** Follow the Kubernetes documentation and setup image provenance. - -### 5.6 Omitted -The v1.5.1 Benchmark skips 5.6 and goes from 5.5 to 5.7. We are including it here merely for explanation. +## 5.7 General Policies +### 5.7.1 Create administrative boundaries between resources using namespaces (Manual) -### 5.7 General Policies -These policies relate to general cluster management topics, like namespace best practices and policies applied to pod objects in the cluster. +**Result:** warn + +**Remediation:** +Follow the documentation and create namespaces for objects in your deployment as you need +them. + +### 5.7.2 Ensure that the seccomp profile is set to docker/default in your pod definitions (Manual) -#### 5.7.1 -Create administrative boundaries between resources using namespaces (Not Scored) -
-Rationale -Limiting the scope of user permissions can reduce the impact of mistakes or malicious activities. A Kubernetes namespace allows you to partition created resources into logically named groups. Resources created in one namespace can be hidden from other namespaces. By default, each resource created by a user in Kubernetes cluster runs in a default namespace, called default. You can create additional namespaces and attach resources and users to them. You can use Kubernetes Authorization plugins to create policies that segregate access to namespace resources between different users. -
+**Result:** warn -**Result:** Not Scored +**Remediation:** +Seccomp is an alpha feature currently. By default, all alpha features are disabled. So, you +would need to enable alpha features in the apiserver by passing "--feature- +gates=AllAlpha=true" argument. +Edit the /etc/kubernetes/apiserver file on the master node and set the KUBE_API_ARGS +parameter to "--feature-gates=AllAlpha=true" +KUBE_API_ARGS="--feature-gates=AllAlpha=true" +Based on your system, restart the kube-apiserver service. For example: +systemctl restart kube-apiserver.service +Use annotations to enable the docker/default seccomp profile in your pod definitions. An +example is as below: +apiVersion: v1 +kind: Pod +metadata: + name: trustworthy-pod + annotations: + seccomp.security.alpha.kubernetes.io/pod: docker/default +spec: + containers: + - name: trustworthy-container + image: sotrustworthy:latest + +### 5.7.3 Apply Security Context to Your Pods and Containers (Manual) + + +**Result:** warn + +**Remediation:** +Follow the Kubernetes documentation and apply security contexts to your pods. For a +suggested list of security contexts, you may refer to the CIS Security Benchmark for Docker +Containers. + +### 5.7.4 The default namespace should not be used (Manual) + + +**Result:** pass + +**Remediation:** +Ensure that namespaces are created to allow for appropriate segregation of Kubernetes +resources and that all new resources are created in a specific namespace. **Audit:** -Run the below command and review the namespaces created in the cluster. ```bash -kubectl get namespaces +kubectl get all --no-headers -n default | grep -v service | wc -l | xargs -I {} echo '--count={}' ``` -Ensure that these namespaces are the ones you need and are adequately administered as per your requirements. +**Expected Result**: -**Remediation:** -Follow the documentation and create namespaces for objects in your deployment as you need them. - - -#### 5.7.2 -Ensure that the seccomp profile is set to `docker/default` in your pod definitions (Not Scored) -
-Rationale -Seccomp (secure computing mode) is used to restrict the set of system calls applications can make, allowing cluster administrators greater control over the security of workloads running in the cluster. Kubernetes disables seccomp profiles by default for historical reasons. You should enable it to ensure that the workloads have restricted actions available within the container. -
- -**Result:** Not Scored - -**Audit:** -Review the pod definitions in your cluster. It should create a line as below: - -```yaml -annotations: - seccomp.security.alpha.kubernetes.io/pod: docker/default +```console +'0' is equal to '0' ``` -**Remediation:** -Review the Kubernetes documentation and if needed, apply a relevant PodSecurityPolicy. +**Returned Value**: -#### 5.7.3 -Apply Security Context to Your Pods and Containers (Not Scored) -
-Rationale -A security context defines the operating system security settings (uid, gid, capabilities, SELinux role, etc..) applied to a container. When designing your containers and pods, make sure that you configure the security context for your pods, containers, and volumes. A security context is a property defined in the deployment yaml. It controls the security parameters that will be assigned to the pod/container/volume. There are two levels of security context: pod level security context, and container-level security context. -
- -**Result:** Not Scored - -**Audit:** -Review the pod definitions in your cluster and verify that you have security contexts defined as appropriate. - -**Remediation:** -Follow the Kubernetes documentation and apply security contexts to your pods. For a suggested list of security contexts, you may refer to the CIS Security Benchmark. - - -#### 5.7.4 -The default namespace should not be used (Scored) -
-Rationale -Resources in a Kubernetes cluster should be segregated by namespace, to allow for security controls to be applied at that level and to make it easier to manage resources. -
- -**Result:** Pass - -**Audit:** -Run the below command on the master node. - -```bash -kubectl get all -n default +```console +--count=0 ``` - -The only entries there should be system-managed resources such as the kubernetes service. - -**Remediation:** -By default, K3s does not utilize the default namespace. diff --git a/content/k3s/latest/en/storage/_index.md b/content/k3s/latest/en/storage/_index.md index fd0dcba1168..883128ee118 100644 --- a/content/k3s/latest/en/storage/_index.md +++ b/content/k3s/latest/en/storage/_index.md @@ -9,6 +9,26 @@ A persistent volume (PV) is a piece of storage in the Kubernetes cluster, while This page describes how to set up persistent storage with a local storage provider, or with [Longhorn.](#setting-up-longhorn) +# What's changed in K3s storage? + +K3s removes several optional volume plugins and all built-in (sometimes referred to as "in-tree") cloud providers. We do this in order to achieve a smaller binary size and to avoid dependence on third-party cloud or data center technologies and services, which may not be available in many K3s use cases. We are able to do this because their removal affects neither core Kubernetes functionality nor conformance. + +The following volume plugins have been removed from K3s: + +* cephfs +* fc +* flocker +* git_repo +* glusterfs +* portworx +* quobyte +* rbd +* storageos + +Both components have out-of-tree alternatives that can be used with K3s: The Kubernetes [Container Storage Interface (CSI)](https://github.com/container-storage-interface/spec/blob/master/spec.md) and [Cloud Provider Interface (CPI)](https://kubernetes.io/docs/tasks/administer-cluster/running-cloud-controller/). + +Kubernetes maintainers are actively migrating in-tree volume plugins to CSI drivers. For more information on this migration, please refer [here](https://kubernetes.io/blog/2021/12/10/storage-in-tree-to-csi-migration-status-update/). + # Setting up the Local Storage Provider K3s comes with Rancher's Local Path Provisioner and this enables the ability to create persistent volume claims out of the box using local storage on the respective node. Below we cover a simple example. For more information please reference the official documentation [here](https://github.com/rancher/local-path-provisioner/blob/master/README.md#usage). diff --git a/content/k3s/latest/en/upgrades/_index.md b/content/k3s/latest/en/upgrades/_index.md index 1b9c86805ad..fad09759854 100644 --- a/content/k3s/latest/en/upgrades/_index.md +++ b/content/k3s/latest/en/upgrades/_index.md @@ -3,12 +3,22 @@ title: "Upgrades" weight: 25 --- -This section describes how to upgrade your K3s cluster. +### Upgrading your K3s cluster [Upgrade basics]({{< baseurl >}}/k3s/latest/en/upgrades/basic/) describes several techniques for upgrading your cluster manually. It can also be used as a basis for upgrading through third-party Infrastructure-as-Code tools like [Terraform](https://www.terraform.io/). [Automated upgrades]({{< baseurl >}}/k3s/latest/en/upgrades/automated/) describes how to perform Kubernetes-native automated upgrades using Rancher's [system-upgrade-controller](https://github.com/rancher/system-upgrade-controller). -> If Traefik is not disabled K3s versions 1.20 and earlier will have installed Traefik v1, while K3s versions 1.21 and later will install Traefik v2 if v1 is not already present. To upgrade Traefik, please refer to the [Traefik documentation](https://doc.traefik.io/traefik/migration/v1-to-v2/) and use the [migration tool](https://github.com/traefik/traefik-migration-tool) to migrate from the older Traefik v1 to Traefik v2. +### Version-specific caveats -> The experimental embedded Dqlite data store was deprecated in K3s v1.19.1. Please note that upgrades from experimental Dqlite to experimental embedded etcd are not supported. If you attempt an upgrade it will not succeed and data will be lost. +- **Traefik:** If Traefik is not disabled, K3s versions 1.20 and earlier will install Traefik v1, while K3s versions 1.21 and later will install Traefik v2, if v1 is not already present. To upgrade from the older Traefik v1 to Traefik v2, please refer to the [Traefik documentation](https://doc.traefik.io/traefik/migration/v1-to-v2/) and use the [migration tool](https://github.com/traefik/traefik-migration-tool). + +- **K3s bootstrap data:** If you are using K3s in an HA configuration with an external SQL datastore, and your server (control-plane) nodes were not started with the `--token` CLI flag, you will no longer be able to add additional K3s servers to the cluster without specifying the token. Ensure that you retain a copy of this token, as it is required when restoring from backup. Previously, K3s did not enforce the use of a token when using external SQL datastores. + - The affected versions are <= v1.19.12+k3s1, v1.20.8+k3s1, v1.21.2+k3s1; the patched versions are v1.19.13+k3s1, v1.20.9+k3s1, v1.21.3+k3s1. + + - You may retrieve the token value from any server already joined to the cluster as follows: +``` +cat /var/lib/rancher/k3s/server/token +``` + +- **Experimental Dqlite:** The experimental embedded Dqlite data store was deprecated in K3s v1.19.1. Please note that upgrades from experimental Dqlite to experimental embedded etcd are not supported. If you attempt an upgrade, it will not succeed, and data will be lost. diff --git a/content/rancher/v2.0-v2.4/_index.md b/content/rancher/v2.0-v2.4/_index.md index 25b54e1aff6..4d23120e33c 100644 --- a/content/rancher/v2.0-v2.4/_index.md +++ b/content/rancher/v2.0-v2.4/_index.md @@ -1,5 +1,5 @@ --- -title: v2.0-v2.4.x +title: Rancher 2.0-2.4 weight: 3 showBreadcrumb: false --- diff --git a/content/rancher/v2.0-v2.4/en/_index.md b/content/rancher/v2.0-v2.4/en/_index.md index 153f22c3b87..6720ef85a73 100644 --- a/content/rancher/v2.0-v2.4/en/_index.md +++ b/content/rancher/v2.0-v2.4/en/_index.md @@ -1,8 +1,8 @@ --- -title: "Rancher v2.0-v2.4" +title: "Rancher 2.0-2.4" shortTitle: "Rancher 2.0-2.4" description: "Rancher adds significant value on top of Kubernetes: managing hundreds of clusters from one interface, centralizing RBAC, enabling monitoring and alerting. Read more." -metaTitle: "Rancher 2.x Docs: What is New?" +metaTitle: "Rancher 2.0-2.4 Docs: What is New?" metaDescription: "Rancher 2 adds significant value on top of Kubernetes: managing hundreds of clusters from one interface, centralizing RBAC, enabling monitoring and alerting. Read more." insertOneSix: true weight: 1 @@ -18,4 +18,4 @@ Rancher adds significant value on top of Kubernetes, first by centralizing authe It then enables detailed monitoring and alerting for clusters and their resources, ships logs to external providers, and integrates directly with Helm via the Application Catalog. If you have an external CI/CD system, you can plug it into Rancher, but if you don't, Rancher even includes a pipeline engine to help you automatically deploy and upgrade workloads. -Rancher is a _complete_ container management platform for Kubernetes, giving you the tools to successfully run Kubernetes anywhere. \ No newline at end of file +Rancher is a _complete_ container management platform for Kubernetes, giving you the tools to successfully run Kubernetes anywhere. diff --git a/content/rancher/v2.0-v2.4/en/admin-settings/authentication/ping-federate/_index.md b/content/rancher/v2.0-v2.4/en/admin-settings/authentication/ping-federate/_index.md index e9a43d062f2..987ab8e966a 100644 --- a/content/rancher/v2.0-v2.4/en/admin-settings/authentication/ping-federate/_index.md +++ b/content/rancher/v2.0-v2.4/en/admin-settings/authentication/ping-federate/_index.md @@ -29,6 +29,8 @@ Note that these URLs will not return valid data until the authentication configu 1. **Groups Field**: Make entries for managing group memberships (example: `memberOf`). + 1. **Entity ID Field** (optional): The published, protocol-dependent, unique identifier of your partner. This ID defines your organization as the entity operating the server for SAML 2.0 transactions. This ID may have been obtained out-of-band or via a SAML metadata file. + 1. **Rancher API Host**: Enter the URL for your Rancher Server. 1. **Private Key** and **Certificate**: This is a key-certificate pair to create a secure shell between Rancher and your IdP. diff --git a/content/rancher/v2.0-v2.4/en/admin-settings/config-private-registry/_index.md b/content/rancher/v2.0-v2.4/en/admin-settings/config-private-registry/_index.md index d258e527c23..09779408bb8 100644 --- a/content/rancher/v2.0-v2.4/en/admin-settings/config-private-registry/_index.md +++ b/content/rancher/v2.0-v2.4/en/admin-settings/config-private-registry/_index.md @@ -4,7 +4,7 @@ weight: 400 aliases: --- -You might want to use a private Docker registry to share your custom base images within your organization. With a private registry, you can keep a private, consistent, and centralized source of truth for the Docker images that are used in your clusters. +You might want to use a private container registry to share your custom base images within your organization. With a private registry, you can keep a private, consistent, and centralized source of truth for the container images that are used in your clusters. There are two main ways to set up private registries in Rancher: by setting up the global default registry through the **Settings** tab in the global view, and by setting up a private registry in the advanced options in the cluster-level settings. The global default registry is intended to be used for air-gapped setups, for registries that do not require credentials. The cluster-level private registry is intended to be used in all setups in which the private registry requires credentials. diff --git a/content/rancher/v2.0-v2.4/en/admin-settings/rbac/cluster-project-roles/_index.md b/content/rancher/v2.0-v2.4/en/admin-settings/rbac/cluster-project-roles/_index.md index 6185daa0bee..6e155df7033 100644 --- a/content/rancher/v2.0-v2.4/en/admin-settings/rbac/cluster-project-roles/_index.md +++ b/content/rancher/v2.0-v2.4/en/admin-settings/rbac/cluster-project-roles/_index.md @@ -84,6 +84,10 @@ _Project roles_ are roles that can be used to grant users access to a project. T These users can manage project-scoped resources like namespaces and workloads, but cannot manage other project members. + >**Note:** + > + >By default, the Rancher role of `project-member` inherits from the `Kubernetes-edit` role, and the `project-owner` role inherits from the `Kubernetes-admin` role. As such, both `project-member` and `project-owner` roles will allow for namespace management, including the ability to create and delete namespaces. + - **Read Only:** These users can view everything in the project but cannot create, update, or delete anything. diff --git a/content/rancher/v2.0-v2.4/en/admin-settings/rbac/global-permissions/_index.md b/content/rancher/v2.0-v2.4/en/admin-settings/rbac/global-permissions/_index.md index 0f5e5f00c9d..125c2cbe699 100644 --- a/content/rancher/v2.0-v2.4/en/admin-settings/rbac/global-permissions/_index.md +++ b/content/rancher/v2.0-v2.4/en/admin-settings/rbac/global-permissions/_index.md @@ -27,25 +27,6 @@ This section covers the following topics: - [Configuring global permissions for groups](#configuring-global-permissions-for-groups) - [Refreshing group memberships](#refreshing-group-memberships) -### List of `restricted-admin` Permissions - -The `restricted-admin` permissions are as follows: - -- Has full admin access to all downstream clusters managed by Rancher. -- Has very limited access to the local Kubernetes cluster. Can access Rancher custom resource definitions, but has no access to any Kubernetes native types. -- Can add other users and assign them to clusters outside of the local cluster. -- Can create other restricted admins. -- Cannot grant any permissions in the local cluster they don't currently have. (This is how Kubernetes normally operates) - - -### Changing Global Administrators to Restricted Admins - -If Rancher already has a global administrator, they should change all global administrators over to the new `restricted-admin` role. - -This can be done through **Security > Users** and moving any Administrator role over to Restricted Administrator. - -Signed-in users can change themselves over to the `restricted-admin` if they wish, but they should only do that as the last step, otherwise they won't have the permissions to do so. - # Global Permission Assignment Global permissions for local users are assigned differently than users who log in to Rancher using external authentication. diff --git a/content/rancher/v2.0-v2.4/en/backups/restore/rke-restore/v2.0-v2.1/_index.md b/content/rancher/v2.0-v2.4/en/backups/restore/rke-restore/v2.0-v2.1/_index.md index b7026acc4a4..bfc20ae2b48 100644 --- a/content/rancher/v2.0-v2.4/en/backups/restore/rke-restore/v2.0-v2.1/_index.md +++ b/content/rancher/v2.0-v2.4/en/backups/restore/rke-restore/v2.0-v2.1/_index.md @@ -28,7 +28,7 @@ Because of the changes necessary to address [CVE-2018-20321](https://cve.mitre.o 2. After executing the command a `tokens.json` file will be created. Important! Back up this file in a safe place.** You will need it to restore functionality to your clusters after rolling back Rancher. **If you lose this file, you may lose access to your clusters.** -3. Rollback Rancher following the [normal instructions]({{}}/rancher/v2.x/en/upgrades/rollbacks/). +3. Rollback Rancher following the [normal instructions]({{}}/rancher/v2.0-v2.4/en/upgrades/rollbacks/). 4. Once Rancher comes back up, every cluster managed by Rancher (except for Imported clusters) will be in an `Unavailable` state. diff --git a/content/rancher/v2.0-v2.4/en/best-practices/management/_index.md b/content/rancher/v2.0-v2.4/en/best-practices/management/_index.md index d85e5a22cbf..5e23755f729 100644 --- a/content/rancher/v2.0-v2.4/en/best-practices/management/_index.md +++ b/content/rancher/v2.0-v2.4/en/best-practices/management/_index.md @@ -4,6 +4,7 @@ weight: 101 aliases: - /rancher/v2.0-v2.4/en/best-practices/management - /rancher/v2.0-v2.4/en/best-practices/v2.0-v2.4/management + - /rancher/v2.x/en/best-practices/management/ - /rancher/v2.x/en/best-practices/v2.0-v2.4/management/ --- @@ -86,8 +87,8 @@ Run your etcd and control plane nodes on virtual machines where you can scale vC ### Use at Least Three etcd Nodes Provision 3 or 5 etcd nodes. Etcd requires a quorum to determine a leader by the majority of nodes, therefore it is not recommended to have clusters of even numbers. Three etcd nodes is generally sufficient for smaller clusters and five etcd nodes for large clusters. -### Use at Least Two Control Plane Nodes -Provision two or more control plane nodes. Some control plane components, such as the `kube-apiserver`, run in [active-active](https://www.jscape.com/blog/active-active-vs-active-passive-high-availability-cluster) mode and will give you more scalability. Other components such as kube-scheduler and kube-controller run in active-passive mode (leader elect) and give you more fault tolerance. +### Use at Least Three Control Plane Nodes +Provision three or more control plane nodes. Some control plane components, such as the `kube-apiserver`, run in [active-active](https://www.jscape.com/blog/active-active-vs-active-passive-high-availability-cluster) mode and will give you more scalability. Other components such as kube-scheduler and kube-controller run in active-passive mode (leader elect) and give you more fault tolerance. ### Monitor Your Cluster Closely monitor and scale your nodes as needed. You should [enable cluster monitoring]({{}}/rancher/v2.0-v2.4/en/monitoring-alerting/legacy/monitoring/cluster-monitoring/) and use the Prometheus metrics and Grafana visualization options as a starting point. diff --git a/content/rancher/v2.0-v2.4/en/cluster-admin/projects-and-namespaces/_index.md b/content/rancher/v2.0-v2.4/en/cluster-admin/projects-and-namespaces/_index.md index 9774a89c470..545de58bf84 100644 --- a/content/rancher/v2.0-v2.4/en/cluster-admin/projects-and-namespaces/_index.md +++ b/content/rancher/v2.0-v2.4/en/cluster-admin/projects-and-namespaces/_index.md @@ -162,6 +162,9 @@ By default, your user is added as the project `Owner`. >**Notes on Permissions:** > >- Users assigned the `Owner` or `Member` role for a project automatically inherit the `namespace creation` role. However, this role is a [Kubernetes ClusterRole](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#role-and-clusterrole), meaning its scope extends to all projects in the cluster. Therefore, users explicitly assigned the `Owner` or `Member` role for a project can create namespaces in other projects they're assigned to, even with only the `Read Only` role assigned. +> +>- By default, the Rancher role of `project-member` inherits from the `Kubernetes-edit` role, and the `project-owner` role inherits from the `Kubernetes-admin` role. As such, both `project-member` and `project-owner` roles will allow for namespace management, including the ability to create and delete namespaces. +> >- Choose `Custom` to create a custom role on the fly: [Custom Project Roles]({{}}/rancher/v2.0-v2.4/en/admin-settings/rbac/cluster-project-roles/#custom-project-roles). To add members: diff --git a/content/rancher/v2.0-v2.4/en/helm-charts/creating-apps/_index.md b/content/rancher/v2.0-v2.4/en/helm-charts/creating-apps/_index.md index a0928f9373a..73d1f6efe48 100644 --- a/content/rancher/v2.0-v2.4/en/helm-charts/creating-apps/_index.md +++ b/content/rancher/v2.0-v2.4/en/helm-charts/creating-apps/_index.md @@ -128,4 +128,4 @@ This reference contains variables that you can use in `questions.yml` nested und # Tutorial: Example Custom Chart Creation -For a tutorial on adding a custom Helm chart to a custom catalog, refer to [this page.]({{}}/rancher/v2.x/en/catalog/tutorial) +For a tutorial on adding a custom Helm chart to a custom catalog, refer to [this page.]({{}}/rancher/v2.0-v2.4/en/catalog/tutorial) diff --git a/content/rancher/v2.0-v2.4/en/helm-charts/tutorial/_index.md b/content/rancher/v2.0-v2.4/en/helm-charts/tutorial/_index.md index 5a23d0b7092..09b5c493fc1 100644 --- a/content/rancher/v2.0-v2.4/en/helm-charts/tutorial/_index.md +++ b/content/rancher/v2.0-v2.4/en/helm-charts/tutorial/_index.md @@ -50,20 +50,20 @@ You can fill your custom catalogs with either Helm Charts or Rancher Charts, alt - CMS questions: - variable: persistence.enabled - default: "false" - description: "Enable persistent volume for WordPress" - type: boolean - required: true - label: WordPress Persistent Volume Enabled - show_subquestion_if: true - group: "WordPress Settings" - subquestions: - - variable: persistence.size + default: "false" + description: "Enable persistent volume for WordPress" + type: boolean + required: true + label: WordPress Persistent Volume Enabled + show_subquestion_if: true + group: "WordPress Settings" + subquestions: + - variable: persistence.size default: "10Gi" description: "WordPress Persistent Volume Size" type: string label: WordPress Volume Size - - variable: persistence.storageClass + - variable: persistence.storageClass default: "" description: "If undefined or null, uses the default StorageClass. Default to null" type: storageclass diff --git a/content/rancher/v2.0-v2.4/en/installation/install-rancher-on-k8s/_index.md b/content/rancher/v2.0-v2.4/en/installation/install-rancher-on-k8s/_index.md index 98c8db4a6e8..0e79ce86a0d 100644 --- a/content/rancher/v2.0-v2.4/en/installation/install-rancher-on-k8s/_index.md +++ b/content/rancher/v2.0-v2.4/en/installation/install-rancher-on-k8s/_index.md @@ -196,7 +196,7 @@ helm install rancher rancher-/rancher \ --namespace cattle-system \ --set hostname=rancher.my.org \ --set ingress.tls.source=letsEncrypt \ - --set letsEncrypt.email=me@example.org + --set letsEncrypt.email=me@example.org \ ``` Wait for Rancher to be rolled out: @@ -231,7 +231,7 @@ helm install rancher rancher-/rancher \ If you are using a Private CA signed certificate , add `--set privateCA=true` to the command: ``` -helm install rancher rancher-latest/rancher \ +helm install rancher rancher-/rancher \ --namespace cattle-system \ --set hostname=rancher.my.org \ --set ingress.tls.source=secret \ diff --git a/content/rancher/v2.0-v2.4/en/installation/install-rancher-on-k8s/chart-options/_index.md b/content/rancher/v2.0-v2.4/en/installation/install-rancher-on-k8s/chart-options/_index.md index ff6631bef9a..f027000a6cb 100644 --- a/content/rancher/v2.0-v2.4/en/installation/install-rancher-on-k8s/chart-options/_index.md +++ b/content/rancher/v2.0-v2.4/en/installation/install-rancher-on-k8s/chart-options/_index.md @@ -126,7 +126,7 @@ To customize or use a different ingress with Rancher server you can set your own Example on setting a custom certificate issuer: ```plain ---set ingress.extraAnnotations.'certmanager\.k8s\.io/cluster-issuer'=ca-key-pair +--set ingress.extraAnnotations.'cert-manager\.io/cluster-issuer'=issuer-name ``` Example on setting a static proxy header with `ingress.configurationSnippet`. This value is parsed like a template so variables can be used. diff --git a/content/rancher/v2.0-v2.4/en/installation/install-rancher-on-k8s/upgrades/_index.md b/content/rancher/v2.0-v2.4/en/installation/install-rancher-on-k8s/upgrades/_index.md index e49c62f7d23..cb3cf8b9655 100644 --- a/content/rancher/v2.0-v2.4/en/installation/install-rancher-on-k8s/upgrades/_index.md +++ b/content/rancher/v2.0-v2.4/en/installation/install-rancher-on-k8s/upgrades/_index.md @@ -168,7 +168,7 @@ helm upgrade rancher rancher-/rancher \ ### Option B: Reinstalling Rancher and cert-manager -If you are currently running the cert-manger whose version is older than v0.11, and want to upgrade both Rancher and cert-manager to a newer version, then you need to reinstall both Rancher and cert-manger due to the API change in cert-manger v0.11. +If you are currently running the cert-manager whose version is older than v0.11, and want to upgrade both Rancher and cert-manager to a newer version, then you need to reinstall both Rancher and cert-manager due to the API change in cert-manager v0.11. 1. Uninstall Rancher diff --git a/content/rancher/v2.0-v2.4/en/installation/install-rancher-on-k8s/upgrades/helm2/_index.md b/content/rancher/v2.0-v2.4/en/installation/install-rancher-on-k8s/upgrades/helm2/_index.md index 7895c69d7af..e0f9ac2787c 100644 --- a/content/rancher/v2.0-v2.4/en/installation/install-rancher-on-k8s/upgrades/helm2/_index.md +++ b/content/rancher/v2.0-v2.4/en/installation/install-rancher-on-k8s/upgrades/helm2/_index.md @@ -114,7 +114,7 @@ helm upgrade --install rancher rancher-/rancher \ {{% accordion label="Option B: Reinstalling Rancher chart" %}} -If you are currently running the cert-manger whose version is older than v0.11, and want to upgrade both Rancher and cert-manager to a newer version, then you need to reinstall both Rancher and cert-manger due to the API change in cert-manger v0.11. +If you are currently running the cert-manager whose version is older than v0.11, and want to upgrade both Rancher and cert-manager to a newer version, then you need to reinstall both Rancher and cert-manager due to the API change in cert-manager v0.11. 1. Uninstall Rancher diff --git a/content/rancher/v2.0-v2.4/en/installation/other-installation-methods/air-gap/prepare-nodes/_index.md b/content/rancher/v2.0-v2.4/en/installation/other-installation-methods/air-gap/prepare-nodes/_index.md index d3a7a28d33d..efd93d093c7 100644 --- a/content/rancher/v2.0-v2.4/en/installation/other-installation-methods/air-gap/prepare-nodes/_index.md +++ b/content/rancher/v2.0-v2.4/en/installation/other-installation-methods/air-gap/prepare-nodes/_index.md @@ -142,7 +142,7 @@ For a how-to guide for setting up a DNS record to route domain traffic to an Ama Rancher supports air gap installs using a secure Docker private registry. You must have your own private registry or other means of distributing Docker images to your machines. -In a later step, when you set up your K3s Kubernetes cluster, you will create a [private registries configuration file]({{}}/k3s/latest/en/installation/private-registry/) with details from this registry. +In a later step, when you set up your RKE Kubernetes cluster, you will create a [private registries configuration file]({{}}/rke/latest/en/config-options/private-registries/) with details from this registry. If you need help with creating a private registry, please refer to the [official Docker documentation.](https://docs.docker.com/registry/deploying/#run-an-externally-accessible-registry) @@ -164,8 +164,6 @@ For an example of one way to set up Linux nodes, refer to this [tutorial]({{}}/k3s/latest/en/installation/private-registry/) with details from this registry. - If you need help with creating a private registry, please refer to the [official Docker documentation.](https://docs.docker.com/registry/) {{% /tab %}} diff --git a/content/rancher/v2.0-v2.4/en/installation/other-installation-methods/single-node-docker/single-node-upgrades/_index.md b/content/rancher/v2.0-v2.4/en/installation/other-installation-methods/single-node-docker/single-node-upgrades/_index.md index be69fcd235e..7e7e935097c 100644 --- a/content/rancher/v2.0-v2.4/en/installation/other-installation-methods/single-node-docker/single-node-upgrades/_index.md +++ b/content/rancher/v2.0-v2.4/en/installation/other-installation-methods/single-node-docker/single-node-upgrades/_index.md @@ -86,7 +86,7 @@ During upgrade, you create a copy of the data from your current Rancher containe ``` - docker run --volumes-from rancher-data -v $PWD:/backup busybox tar zcvf /backup/rancher-data-backup--.tar.gz /var/lib/rancher + docker run --volumes-from rancher-data -v "$PWD:/backup" --rm busybox tar zcvf /backup/rancher-data-backup--.tar.gz /var/lib/rancher ``` **Step Result:** When you enter this command, a series of commands should run. diff --git a/content/rancher/v2.0-v2.4/en/installation/requirements/_index.md b/content/rancher/v2.0-v2.4/en/installation/requirements/_index.md index ef77bd8dd89..fbf442d663e 100644 --- a/content/rancher/v2.0-v2.4/en/installation/requirements/_index.md +++ b/content/rancher/v2.0-v2.4/en/installation/requirements/_index.md @@ -78,7 +78,8 @@ Performance increased in Rancher v2.4.0. For the requirements of Rancher before | X-Large | Up to 1000 | Up to 10,000 | 16 | 64 GB | | XX-Large | Up to 2000 | Up to 20,000 | 32 | 128 GB | -[Contact Rancher](https://rancher.com/contact/) for more than 2000 clusters and/or 20,000 nodes. +Every use case and environment is different. Please [contact Rancher](https://rancher.com/contact/) to review yours. + {{% /tab %}} {{% tab "K3s" %}} @@ -93,7 +94,7 @@ These requirements apply to each host in a [K3s Kubernetes cluster where the Ran | X-Large | Up to 1000 | Up to 10,000 | 16 | 64 GB | 2 cores, 4 GB + 1000 IOPS | | XX-Large | Up to 2000 | Up to 20,000 | 32 | 128 GB | 2 cores, 4 GB + 1000 IOPS | -[Contact Rancher](https://rancher.com/contact/) for more than 2000 clusters and/or 20,000 nodes. +Every use case and environment is different. Please [contact Rancher](https://rancher.com/contact/) to review yours. {{% /tab %}} diff --git a/content/rancher/v2.0-v2.4/en/installation/requirements/installing-docker/_index.md b/content/rancher/v2.0-v2.4/en/installation/requirements/installing-docker/_index.md index 4414cb08794..9fdeb407a47 100644 --- a/content/rancher/v2.0-v2.4/en/installation/requirements/installing-docker/_index.md +++ b/content/rancher/v2.0-v2.4/en/installation/requirements/installing-docker/_index.md @@ -3,7 +3,7 @@ title: Installing Docker weight: 1 --- -For Helm CLI installs, Docker is required to be installed on any node that runs the Rancher server. +Docker is required to be installed on nodes where the Rancher server will be installed with Helm or Docker. There are a couple of options for installing Docker. One option is to refer to the [official Docker documentation](https://docs.docker.com/install/) about how to install Docker on Linux. The steps will vary based on the Linux distribution. diff --git a/content/rancher/v2.0-v2.4/en/installation/resources/advanced/firewall/_index.md b/content/rancher/v2.0-v2.4/en/installation/resources/advanced/firewall/_index.md index f3ee9defadd..67c6f880325 100644 --- a/content/rancher/v2.0-v2.4/en/installation/resources/advanced/firewall/_index.md +++ b/content/rancher/v2.0-v2.4/en/installation/resources/advanced/firewall/_index.md @@ -3,7 +3,7 @@ title: Opening Ports with firewalld weight: 1 --- -> We recommend disabling firewalld. For Kubernetes 1.19, firewalld must be turned off. +> We recommend disabling firewalld. For Kubernetes 1.19.x and higher, firewalld must be turned off. Some distributions of Linux [derived from RHEL,](https://en.wikipedia.org/wiki/Red_Hat_Enterprise_Linux#Rebuilds) including Oracle Linux, may have default firewall rules that block communication with Helm. diff --git a/content/rancher/v2.0-v2.4/en/installation/resources/advanced/helm2/helm-rancher/_index.md b/content/rancher/v2.0-v2.4/en/installation/resources/advanced/helm2/helm-rancher/_index.md index 985141b076a..dc96db26d41 100644 --- a/content/rancher/v2.0-v2.4/en/installation/resources/advanced/helm2/helm-rancher/_index.md +++ b/content/rancher/v2.0-v2.4/en/installation/resources/advanced/helm2/helm-rancher/_index.md @@ -131,7 +131,12 @@ deployment "rancher" successfully rolled out This option uses `cert-manager` to automatically request and renew [Let's Encrypt](https://letsencrypt.org/) certificates. This is a free service that provides you with a valid certificate as Let's Encrypt is a trusted CA. This configuration uses HTTP validation (`HTTP-01`) so the load balancer must have a public DNS record and be accessible from the internet. -- Set `hostname` to the public DNS record, set `ingress.tls.source` to `letsEncrypt` and `letsEncrypt.email` to the email address used for communication about your certificate (for example, expiry notices) +In the following command, + +- Set `hostname` to the public DNS record that resolves to your load balancer. +- Set `ingress.tls.source` to `letsEncrypt`. +- Set `letsEncrypt.email` to the email address used for communication about your certificate (for example, expiry notices). +- Set `letsEncrypt.ingress.class` to whatever your ingress controller is, e.g., `traefik`, `nginx`, `haproxy`, etc. - If you are installing an alpha version, Helm requires adding the `--devel` option to the command. ``` @@ -140,7 +145,8 @@ helm install rancher-/rancher \ --namespace cattle-system \ --set hostname=rancher.my.org \ --set ingress.tls.source=letsEncrypt \ - --set letsEncrypt.email=me@example.org + --set letsEncrypt.email=me@example.org \ + --set letsEncrypt.ingress.class=nginx ``` Wait for Rancher to be rolled out: diff --git a/content/rancher/v2.0-v2.4/en/installation/resources/feature-flags/_index.md b/content/rancher/v2.0-v2.4/en/installation/resources/feature-flags/_index.md index c235116ee24..3c0c500c5aa 100644 --- a/content/rancher/v2.0-v2.4/en/installation/resources/feature-flags/_index.md +++ b/content/rancher/v2.0-v2.4/en/installation/resources/feature-flags/_index.md @@ -107,7 +107,7 @@ When installing Rancher with Docker, use the `--features` option. In the below e docker run -d -p 80:80 -p 443:443 \ --restart=unless-stopped \ rancher/rancher:rancher-latest \ - --features==true,=true # Available as of v2.3.0 + --features==true,=true # Available as of v2.3.0 ``` {{% /tab %}} diff --git a/content/rancher/v2.0-v2.4/en/installation/resources/installing-docker/_index.md b/content/rancher/v2.0-v2.4/en/installation/resources/installing-docker/_index.md deleted file mode 100644 index faec8aed59c..00000000000 --- a/content/rancher/v2.0-v2.4/en/installation/resources/installing-docker/_index.md +++ /dev/null @@ -1,20 +0,0 @@ ---- -title: Installing Docker -weight: 1 -aliases: - - /rancher/v2.0-v2.4/en/installation/requirements/installing-docker ---- - -Docker is required to be installed on nodes where the Rancher server will be installed with Helm or Docker. - -There are a couple of options for installing Docker. One option is to refer to the [official Docker documentation](https://docs.docker.com/install/) about how to install Docker on Linux. The steps will vary based on the Linux distribution. - -Another option is to use one of Rancher's Docker installation scripts, which are available for most recent versions of Docker. - -For example, this command could be used to install Docker 19.03 on Ubuntu: - -``` -curl https://releases.rancher.com/install-docker/19.03.sh | sh -``` - -Rancher has installation scripts for every version of upstream Docker that Kubernetes supports. To find out whether a script is available for installing a certain Docker version, refer to this [GitHub repository,](https://github.com/rancher/install-docker) which contains all of Rancher's Docker installation scripts. \ No newline at end of file diff --git a/content/rancher/v2.0-v2.4/en/installation/resources/tls-secrets/_index.md b/content/rancher/v2.0-v2.4/en/installation/resources/tls-secrets/_index.md index 40f9c5294d2..3c32084c5e6 100644 --- a/content/rancher/v2.0-v2.4/en/installation/resources/tls-secrets/_index.md +++ b/content/rancher/v2.0-v2.4/en/installation/resources/tls-secrets/_index.md @@ -35,4 +35,4 @@ kubectl -n cattle-system create secret generic tls-ca \ # Updating a Private CA Certificate -Follow the steps on [this page]({{}}/rancher/v2.x/en/installation/resources/update-ca-cert) to update the SSL certificate of the ingress in a Rancher [high availability Kubernetes installation]({{}}/rancher/v2.0-v2.4/en/installation/install-rancher-on-k8s/) or to switch from the default self-signed certificate to a custom certificate. \ No newline at end of file +Follow the steps on [this page]({{}}/rancher/v2.0-v2.4/en/installation/resources/update-ca-cert) to update the SSL certificate of the ingress in a Rancher [high availability Kubernetes installation]({{}}/rancher/v2.0-v2.4/en/installation/install-rancher-on-k8s/) or to switch from the default self-signed certificate to a custom certificate. \ No newline at end of file diff --git a/content/rancher/v2.0-v2.4/en/installation/resources/upgrading-cert-manager/_index.md b/content/rancher/v2.0-v2.4/en/installation/resources/upgrading-cert-manager/_index.md index 4d8266873c8..29d2e411443 100644 --- a/content/rancher/v2.0-v2.4/en/installation/resources/upgrading-cert-manager/_index.md +++ b/content/rancher/v2.0-v2.4/en/installation/resources/upgrading-cert-manager/_index.md @@ -19,7 +19,7 @@ To address these changes, this guide will do two things: 1. Explain the cert-manager API changes and link to cert-manager's official documentation for migrating your data > **Important:** -> If you are currently running the cert-manger whose version is older than v0.11, and want to upgrade both Rancher and cert-manager to a newer version, you need to reinstall both of them: +> If you are currently running the cert-manager whose version is older than v0.11, and want to upgrade both Rancher and cert-manager to a newer version, you need to reinstall both of them: > 1. Take a one-time snapshot of your Kubernetes cluster running Rancher server > 2. Uninstall Rancher, cert-manager, and the CustomResourceDefinition for cert-manager diff --git a/content/rancher/v2.0-v2.4/en/installation/resources/upgrading-cert-manager/helm-2-instructions/_index.md b/content/rancher/v2.0-v2.4/en/installation/resources/upgrading-cert-manager/helm-2-instructions/_index.md index aba42eb1959..968cd6d6666 100644 --- a/content/rancher/v2.0-v2.4/en/installation/resources/upgrading-cert-manager/helm-2-instructions/_index.md +++ b/content/rancher/v2.0-v2.4/en/installation/resources/upgrading-cert-manager/helm-2-instructions/_index.md @@ -19,7 +19,7 @@ To address these changes, this guide will do two things: 1. Explain the cert-manager API changes and link to cert-manager's offficial documentation for migrating your data > **Important:** -> If you are currently running the cert-manger whose version is older than v0.11, and want to upgrade both Rancher and cert-manager to a newer version, you need to reinstall both of them: +> If you are currently running the cert-manager whose version is older than v0.11, and want to upgrade both Rancher and cert-manager to a newer version, you need to reinstall both of them: > 1. Take a one-time snapshot of your Kubernetes cluster running Rancher server > 2. Uninstall Rancher, cert-manager, and the CustomResourceDefinition for cert-manager diff --git a/content/rancher/v2.0-v2.4/en/k8s-in-rancher/workloads/deploy-workloads/_index.md b/content/rancher/v2.0-v2.4/en/k8s-in-rancher/workloads/deploy-workloads/_index.md index 95a39db870d..249bd6e59d9 100644 --- a/content/rancher/v2.0-v2.4/en/k8s-in-rancher/workloads/deploy-workloads/_index.md +++ b/content/rancher/v2.0-v2.4/en/k8s-in-rancher/workloads/deploy-workloads/_index.md @@ -15,7 +15,7 @@ Deploy a workload to run an application in one or more containers. 1. Enter a **Name** for the workload. -1. Select a [workload type]({{}}/rancher/v2.0-v2.4/en/k8s-in-rancher/workloads/). The workload defaults to a scalable deployment, by can change the workload type by clicking **More options.** +1. Select a [workload type]({{}}/rancher/v2.0-v2.4/en/k8s-in-rancher/workloads/). The workload defaults to a scalable deployment, but you can change the workload type by clicking **More options.** 1. From the **Docker Image** field, enter the name of the Docker image that you want to deploy to the project, optionally prefacing it with the registry host (e.g. `quay.io`, `registry.gitlab.com`, etc.). During deployment, Rancher pulls this image from the specified public or private registry. If no registry host is provided, Rancher will pull the image from [Docker Hub](https://hub.docker.com/explore/). Enter the name exactly as it appears in the registry server, including any required path, and optionally including the desired tag (e.g. `registry.gitlab.com/user/path/image:tag`). If no tag is provided, the `latest` tag will be automatically used. diff --git a/content/rancher/v2.0-v2.4/en/overview/_index.md b/content/rancher/v2.0-v2.4/en/overview/_index.md index b572069fbbb..16d8a5d0590 100644 --- a/content/rancher/v2.0-v2.4/en/overview/_index.md +++ b/content/rancher/v2.0-v2.4/en/overview/_index.md @@ -37,7 +37,7 @@ The Rancher API server is built on top of an embedded Kubernetes API server and - **Provisioning Kubernetes clusters:** The Rancher API server can [provision Kubernetes]({{}}/rancher/v2.0-v2.4/en/cluster-provisioning/) on existing nodes, or perform [Kubernetes upgrades.]({{}}/rancher/v2.0-v2.4/en/cluster-admin/upgrading-kubernetes) - **Catalog management:** Rancher provides the ability to use a [catalog of Helm charts]({{}}/rancher/v2.0-v2.4/en/catalog/) that make it easy to repeatedly deploy applications. -- **Managing projects:** A project is a group of multiple namespaces and access control policies within a cluster. A project is a Rancher concept, not a Kubernetes concept, which allows you manage multiple namespaces as a group and perform Kubernetes operations in them. The Rancher UI provides features for [project administration]({{}}/rancher/v2.0-v2.4/en/project-admin/) and for [managing applications within projects.]({{}}/rancher/v2.0-v2.4/en/k8s-in-rancher/) +- **Managing projects:** A project is a group of multiple namespaces and access control policies within a cluster. A project is a Rancher concept, not a Kubernetes concept, which allows you to manage multiple namespaces as a group and perform Kubernetes operations in them. The Rancher UI provides features for [project administration]({{}}/rancher/v2.0-v2.4/en/project-admin/) and for [managing applications within projects.]({{}}/rancher/v2.0-v2.4/en/k8s-in-rancher/) - **Pipelines:** Setting up a [pipeline]({{}}/rancher/v2.0-v2.4/en/project-admin/pipelines/) can help developers deliver new software as quickly and efficiently as possible. Within Rancher, you can configure pipelines for each of your Rancher projects. - **Istio:** Our [integration with Istio]({{}}/rancher/v2.0-v2.4/en/cluster-admin/tools/istio/) is designed so that a Rancher operator, such as an administrator or cluster owner, can deliver Istio to developers. Then developers can use Istio to enforce security policies, troubleshoot problems, or manage traffic for green/blue deployments, canary deployments, or A/B testing. diff --git a/content/rancher/v2.0-v2.4/en/overview/concepts/_index.md b/content/rancher/v2.0-v2.4/en/overview/concepts/_index.md index afac8412a3f..c637928995d 100644 --- a/content/rancher/v2.0-v2.4/en/overview/concepts/_index.md +++ b/content/rancher/v2.0-v2.4/en/overview/concepts/_index.md @@ -52,7 +52,7 @@ Three etcd nodes is generally sufficient for smaller clusters and five etcd node ### Controlplane Nodes -Controlplane nodes run the Kubernetes API server, scheduler, and controller manager. These nodes take care of routine tasks to ensure that your cluster maintains your configuration. Because all cluster data is stored on your etcd nodes, control plane nodes are stateless. You can run control plane on a single node, although two or more nodes are recommended for redundancy. Additionally, a single node can share the control plane and etcd roles. +Controlplane nodes run the Kubernetes API server, scheduler, and controller manager. These nodes take care of routine tasks to ensure that your cluster maintains your configuration. Because all cluster data is stored on your etcd nodes, control plane nodes are stateless. You can run control plane on a single node, although three or more nodes are recommended for redundancy. Additionally, a single node can share the control plane and etcd roles. ### Worker Nodes @@ -69,4 +69,4 @@ For high-availability installations of Rancher, Helm is the tool used to install Helm is the package management tool of choice for Kubernetes. Helm charts provide templating syntax for Kubernetes YAML manifest documents. With Helm we can create configurable deployments instead of just using static files. For more information about creating your own catalog of deployments, check out the docs at [https://helm.sh/](https://helm.sh). -For more information on service accounts and cluster role binding, refer to the [Kubernetes documentation.](https://kubernetes.io/docs/reference/access-authn-authz/rbac/) \ No newline at end of file +For more information on service accounts and cluster role binding, refer to the [Kubernetes documentation.](https://kubernetes.io/docs/reference/access-authn-authz/rbac/) diff --git a/content/rancher/v2.0-v2.4/en/project-admin/project-members/_index.md b/content/rancher/v2.0-v2.4/en/project-admin/project-members/_index.md index 47f4c7eddfe..a0a4a0922d7 100644 --- a/content/rancher/v2.0-v2.4/en/project-admin/project-members/_index.md +++ b/content/rancher/v2.0-v2.4/en/project-admin/project-members/_index.md @@ -42,6 +42,8 @@ Following project creation, you can add users as project members so that they ca > >- Users assigned the `Owner` or `Member` role for a project automatically inherit the `namespace creation` role. However, this role is a [Kubernetes ClusterRole](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#role-and-clusterrole), meaning its scope extends to all projects in the cluster. Therefore, users explicitly assigned the `Owner` or `Member` role for a project can create namespaces in other projects they're assigned to, even with only the `Read Only` role assigned. > + >- By default, the Rancher role of `project-member` inherits from the `Kubernetes-edit` role, and the `project-owner` role inherits from the `Kubernetes-admin` role. As such, both `project-member` and `project-owner` roles will allow for namespace management, including the ability to create and delete namespaces. + > >- For `Custom` roles, you can modify the list of individual roles available for assignment. > > - To add roles to the list, [Add a Custom Role]({{}}/rancher/v2.0-v2.4/en/admin-settings/rbac/default-custom-roles). diff --git a/content/rancher/v2.0-v2.4/en/project-admin/resource-quotas/_index.md b/content/rancher/v2.0-v2.4/en/project-admin/resource-quotas/_index.md index 6cac538d8f9..af27d4b3457 100644 --- a/content/rancher/v2.0-v2.4/en/project-admin/resource-quotas/_index.md +++ b/content/rancher/v2.0-v2.4/en/project-admin/resource-quotas/_index.md @@ -45,4 +45,4 @@ Edit [resource quotas]({{}}/rancher/v2.0-v2.4/en/k8s-in-rancher/project 1. Click **Create**. -**Result:** The resource quota is applied to your project and namespaces. When you add more namespaces in the future, Rancher validates that the project can accommodate the namespace. If the project can't allocate the resources, Rancher won't let you save your changes. +**Result:** The resource quota is applied to your project and namespaces. When you add more namespaces in the future, Rancher validates that the project can accommodate the namespace. If the project can't allocate the resources, you may still create namespaces, but they will be given a resource quota of 0. Subsequently, Rancher will not allow you to create any resources restricted by this quota. diff --git a/content/rancher/v2.0-v2.4/en/project-admin/tools/_index.md b/content/rancher/v2.0-v2.4/en/project-admin/tools/_index.md index bdd701e0d31..c1adfb7bc85 100644 --- a/content/rancher/v2.0-v2.4/en/project-admin/tools/_index.md +++ b/content/rancher/v2.0-v2.4/en/project-admin/tools/_index.md @@ -43,4 +43,4 @@ For details on project-level logging, see [this section.](./project-logging) _Available as of v2.2.0_ -Using Rancher, you can monitor the state and processes of your cluster nodes, Kubernetes components, and software deployments through integration with [Prometheus](https://prometheus.io/), a leading open-source monitoring solution. For details, refer to the [monitoring section.]({{}}/rancher/v2.0-v2.4/en/cluster-admin/tools/monitoring) +Using Rancher, you can monitor the state and processes of your cluster nodes, Kubernetes components, and software deployments through integration with [Prometheus](https://prometheus.io/), a leading open-source monitoring solution. For details, refer to the [monitoring section.]({{}}/rancher/v2.0-v2.4/en/cluster-admin/tools/cluster-monitoring) diff --git a/content/rancher/v2.0-v2.4/en/quick-start-guide/deployment/amazon-aws-qs/_index.md b/content/rancher/v2.0-v2.4/en/quick-start-guide/deployment/amazon-aws-qs/_index.md index 9c45743601b..92e07f38fc3 100644 --- a/content/rancher/v2.0-v2.4/en/quick-start-guide/deployment/amazon-aws-qs/_index.md +++ b/content/rancher/v2.0-v2.4/en/quick-start-guide/deployment/amazon-aws-qs/_index.md @@ -5,6 +5,8 @@ weight: 100 --- The following steps will quickly deploy a Rancher Server on AWS with a single node cluster attached. +>**Note:** The intent of these guides is to quickly launch a sandbox that you can use to evaluate Rancher. These guides are not intended for production environments. For comprehensive setup instructions, see [Installation]({{}}/rancher/v2.0-v2.4/en/installation/). + ## Prerequisites >**Note** diff --git a/content/rancher/v2.0-v2.4/en/quick-start-guide/deployment/digital-ocean-qs/_index.md b/content/rancher/v2.0-v2.4/en/quick-start-guide/deployment/digital-ocean-qs/_index.md index 0f11b237ccf..06f557f5c88 100644 --- a/content/rancher/v2.0-v2.4/en/quick-start-guide/deployment/digital-ocean-qs/_index.md +++ b/content/rancher/v2.0-v2.4/en/quick-start-guide/deployment/digital-ocean-qs/_index.md @@ -5,6 +5,8 @@ weight: 100 --- The following steps will quickly deploy a Rancher Server on DigitalOcean with a single node cluster attached. +>**Note:** The intent of these guides is to quickly launch a sandbox that you can use to evaluate Rancher. These guides are not intended for production environments. For comprehensive setup instructions, see [Installation]({{}}/rancher/v2.0-v2.4/en/installation/). + ## Prerequisites >**Note** diff --git a/content/rancher/v2.0-v2.4/en/quick-start-guide/deployment/google-gcp-qs/_index.md b/content/rancher/v2.0-v2.4/en/quick-start-guide/deployment/google-gcp-qs/_index.md index 0eeb4a5624d..31911cc0af8 100644 --- a/content/rancher/v2.0-v2.4/en/quick-start-guide/deployment/google-gcp-qs/_index.md +++ b/content/rancher/v2.0-v2.4/en/quick-start-guide/deployment/google-gcp-qs/_index.md @@ -5,6 +5,8 @@ weight: 100 --- The following steps will quickly deploy a Rancher server on GCP in a single-node RKE Kubernetes cluster, with a single-node downstream Kubernetes cluster attached. +>**Note:** The intent of these guides is to quickly launch a sandbox that you can use to evaluate Rancher. These guides are not intended for production environments. For comprehensive setup instructions, see [Installation]({{}}/rancher/v2.0-v2.4/en/installation/). + ## Prerequisites >**Note** diff --git a/content/rancher/v2.0-v2.4/en/quick-start-guide/deployment/microsoft-azure-qs/_index.md b/content/rancher/v2.0-v2.4/en/quick-start-guide/deployment/microsoft-azure-qs/_index.md index 663c03edc1b..448549c6778 100644 --- a/content/rancher/v2.0-v2.4/en/quick-start-guide/deployment/microsoft-azure-qs/_index.md +++ b/content/rancher/v2.0-v2.4/en/quick-start-guide/deployment/microsoft-azure-qs/_index.md @@ -6,6 +6,8 @@ weight: 100 The following steps will quickly deploy a Rancher server on Azure in a single-node RKE Kubernetes cluster, with a single-node downstream Kubernetes cluster attached. +>**Note:** The intent of these guides is to quickly launch a sandbox that you can use to evaluate Rancher. These guides are not intended for production environments. For comprehensive setup instructions, see [Installation]({{}}/rancher/v2.0-v2.4/en/installation/). + ## Prerequisites >**Note** diff --git a/content/rancher/v2.0-v2.4/en/quick-start-guide/deployment/quickstart-manual-setup/_index.md b/content/rancher/v2.0-v2.4/en/quick-start-guide/deployment/quickstart-manual-setup/_index.md index 33d31c1b52c..753006eb6b6 100644 --- a/content/rancher/v2.0-v2.4/en/quick-start-guide/deployment/quickstart-manual-setup/_index.md +++ b/content/rancher/v2.0-v2.4/en/quick-start-guide/deployment/quickstart-manual-setup/_index.md @@ -8,6 +8,8 @@ Howdy Partner! This tutorial walks you through: - Creation of your first cluster - Deployment of an application, Nginx +>**Note:** The intent of these guides is to quickly launch a sandbox that you can use to evaluate Rancher. These guides are not intended for production environments. For comprehensive setup instructions, see [Installation]({{}}/rancher/v2.0-v2.4/en/installation/). + ## Quick Start Outline This Quick Start Guide is divided into different tasks for easier consumption. @@ -96,13 +98,13 @@ In this task, you can use the versatile **Custom** option. This option lets you 11. When you finish running the command on your Linux host, click **Done**. -**Result:** +**Result:** Your cluster is created and assigned a state of **Provisioning.** Rancher is standing up your cluster. You can access your cluster after its state is updated to **Active.** -**Active** clusters are assigned two Projects: +**Active** clusters are assigned two Projects: - `Default`, containing the `default` namespace - `System`, containing the `cattle-system`, `ingress-nginx`, `kube-public`, and `kube-system` namespaces diff --git a/content/rancher/v2.0-v2.4/en/quick-start-guide/deployment/quickstart-vagrant/_index.md b/content/rancher/v2.0-v2.4/en/quick-start-guide/deployment/quickstart-vagrant/_index.md index 933a94ce66b..a996135f282 100644 --- a/content/rancher/v2.0-v2.4/en/quick-start-guide/deployment/quickstart-vagrant/_index.md +++ b/content/rancher/v2.0-v2.4/en/quick-start-guide/deployment/quickstart-vagrant/_index.md @@ -4,6 +4,8 @@ weight: 200 --- The following steps quickly deploy a Rancher Server with a single node cluster attached. +>**Note:** The intent of these guides is to quickly launch a sandbox that you can use to evaluate Rancher. These guides are not intended for production environments. For comprehensive setup instructions, see [Installation]({{}}/rancher/v2.0-v2.4/en/installation/). + ## Prerequisites - [Vagrant](https://www.vagrantup.com): Vagrant is required as this is used to provision the machine based on the Vagrantfile. @@ -14,7 +16,7 @@ The following steps quickly deploy a Rancher Server with a single node cluster a - Vagrant will require plugins to create VirtualBox VMs. Install them with the following commands: `vagrant plugin install vagrant-vboxmanage` - + `vagrant plugin install vagrant-vbguest` ## Getting Started diff --git a/content/rancher/v2.0-v2.4/en/troubleshooting/dns/_index.md b/content/rancher/v2.0-v2.4/en/troubleshooting/dns/_index.md index 48a98a37919..169b5d84104 100644 --- a/content/rancher/v2.0-v2.4/en/troubleshooting/dns/_index.md +++ b/content/rancher/v2.0-v2.4/en/troubleshooting/dns/_index.md @@ -158,7 +158,7 @@ kubectl run -i --restart=Never --rm test-${RANDOM} --image=ubuntu --overrides='{ Enabling query logging can be done by enabling the [log plugin](https://coredns.io/plugins/log/) in the Corefile configuration in the configmap `coredns`. You can do so by using `kubectl -n kube-system edit configmap coredns` or use the command below to replace the configuration in place: ``` -kubectl get configmap -n kube-system coredns -o json | kubectl get configmap -n kube-system coredns -o json | sed -e 's_loadbalance_log\\n loadbalance_g' | kubectl apply -f - +kubectl get configmap -n kube-system coredns -o json | sed -e 's_loadbalance_log\\n loadbalance_g' | kubectl apply -f - ``` All queries will now be logged and can be checked using the command in [Check CoreDNS logging](#check-coredns-logging). diff --git a/content/rancher/v2.5/_index.md b/content/rancher/v2.5/_index.md index 89c7c0d8e09..61f266de705 100644 --- a/content/rancher/v2.5/_index.md +++ b/content/rancher/v2.5/_index.md @@ -1,5 +1,5 @@ --- -title: Rancher 2.5.7-2.5.9 +title: Rancher 2.5 weight: 2 showBreadcrumb: false --- diff --git a/content/rancher/v2.5/en/_index.md b/content/rancher/v2.5/en/_index.md index 513fb4d9836..95a369285c1 100644 --- a/content/rancher/v2.5/en/_index.md +++ b/content/rancher/v2.5/en/_index.md @@ -2,7 +2,7 @@ title: "Rancher 2.5" shortTitle: "Rancher 2.5" description: "Rancher adds significant value on top of Kubernetes: managing hundreds of clusters from one interface, centralizing RBAC, enabling monitoring and alerting. Read more." -metaTitle: "Rancher 2.5.7-2.5.9 Docs: What is New?" +metaTitle: "Rancher 2.5 Docs: What is New?" metaDescription: "Rancher 2 adds significant value on top of Kubernetes: managing hundreds of clusters from one interface, centralizing RBAC, enabling monitoring and alerting. Read more." insertOneSix: false weight: 2 diff --git a/content/rancher/v2.5/en/admin-settings/authentication/ad/_index.md b/content/rancher/v2.5/en/admin-settings/authentication/ad/_index.md index 0cce01d449d..2c9c82a4374 100644 --- a/content/rancher/v2.5/en/admin-settings/authentication/ad/_index.md +++ b/content/rancher/v2.5/en/admin-settings/authentication/ad/_index.md @@ -101,7 +101,7 @@ The table below details the parameters for the group schema configuration. | Search Attribute | Attribute used to construct search filters when adding groups to clusters or projects. See description of user schema `Search Attribute`. | | Search Filter | This filter gets applied to the list of groups that is searched when Rancher attempts to add groups to a site access list or tries to add groups to clusters or projects. For example, a group search filter could be (|(cn=group1)(cn=group2)). Note: If the search filter does not use [valid AD search syntax,](https://docs.microsoft.com/en-us/windows/win32/adsi/search-filter-syntax) the list of groups will be empty. | | Group DN Attribute | The name of the group attribute whose format matches the values in the user attribute describing a the user's memberships. See `User Member Attribute`. | -| Nested Group Membership | This settings defines whether Rancher should resolve nested group memberships. Use only if your organisation makes use of these nested memberships (ie. you have groups that contain other groups as members. We advise avoiding nested groups when possible). | +| Nested Group Membership | This settings defines whether Rancher should resolve nested group memberships. Use only if your organization makes use of these nested memberships (i.e., you have groups that contain other groups as members. We advise avoiding nested groups when possible). | --- @@ -153,7 +153,7 @@ This command performs an LDAP search with the search base set to the domain root Since in this case the user's DN is `CN=John Doe,CN=Users,DC=acme,DC=com` [5], we should configure the **User Search Base** with the parent node DN `CN=Users,DC=acme,DC=com`. -Similarly, based on the DN of the group referenced in the **memberOf** attribute [4], the correct value for the **Group Search Base** would be the parent node of that value, ie. `OU=Groups,DC=acme,DC=com`. +Similarly, based on the DN of the group referenced in the **memberOf** attribute [4], the correct value for the **Group Search Base** would be the parent node of that value, i.e., `OU=Groups,DC=acme,DC=com`. ### Identify User Schema @@ -166,7 +166,7 @@ The output of the above `ldapsearch` query also allows to determine the correct > **Note:** > -> If the AD users in our organisation were to authenticate with their UPN (e.g. jdoe@acme.com) instead of the short logon name, then we would have to set the `Login Attribute` to **userPrincipalName** instead. +> If the AD users in our organization were to authenticate with their UPN (e.g. jdoe@acme.com) instead of the short logon name, then we would have to set the `Login Attribute` to **userPrincipalName** instead. We'll also set the `Search Attribute` parameter to **sAMAccountName|name**. That way users can be added to clusters/projects in the Rancher UI either by entering their username or full name. diff --git a/content/rancher/v2.5/en/admin-settings/authentication/keycloak/_index.md b/content/rancher/v2.5/en/admin-settings/authentication/keycloak/_index.md index 95a37c00e2f..e4e75f36477 100644 --- a/content/rancher/v2.5/en/admin-settings/authentication/keycloak/_index.md +++ b/content/rancher/v2.5/en/admin-settings/authentication/keycloak/_index.md @@ -25,36 +25,87 @@ If your organization uses Keycloak Identity Provider (IdP) for user authenticati >1: Optionally, you can enable either one or both of these settings. >2: Rancher SAML metadata won't be generated until a SAML provider is configured and saved. - + {{< img "/img/rancher/keycloak/keycloak-saml-client-configuration.png" "">}} - + - In the new SAML client, create Mappers to expose the users fields - Add all "Builtin Protocol Mappers" {{< img "/img/rancher/keycloak/keycloak-saml-client-builtin-mappers.png" "">}} - Create a new "Group list" mapper to map the member attribute to a user's groups - {{< img "/img/rancher/keycloak/keycloak-saml-client-group-mapper.png" "">}} -- Export a `metadata.xml` file from your Keycloak client: - From the `Installation` tab, choose the `SAML Metadata IDPSSODescriptor` format option and download your file. - - >**Note** - > Keycloak versions 6.0.0 and up no longer provide the IDP metadata under the `Installation` tab. - > You can still get the XML from the following url: - > - > `https://{KEYCLOAK-URL}/auth/realms/{REALM-NAME}/protocol/saml/descriptor` - > - > The XML obtained from this URL contains `EntitiesDescriptor` as the root element. Rancher expects the root element to be `EntityDescriptor` rather than `EntitiesDescriptor`. So before passing this XML to Rancher, follow these steps to adjust it: - > - > * Copy all the attributes from `EntitiesDescriptor` to the `EntityDescriptor` that are not present. - > * Remove the `` tag from the beginning. - > * Remove the `` from the end of the xml. - > - > You are left with something similar as the example below: - > - > ``` - > - > .... - > - > ``` + {{< img "/img/rancher/keycloak/keycloak-saml-client-group-mapper.png" "">}} + +## Getting the IDP Metadata + +{{% tabs %}} +{{% tab "Keycloak 5 and earlier" %}} +To get the IDP metadata, export a `metadata.xml` file from your Keycloak client. +From the **Installation** tab, choose the **SAML Metadata IDPSSODescriptor** format option and download your file. +{{% /tab %}} +{{% tab "Keycloak 6-13" %}} + +1. From the **Configure** section, click the **Realm Settings** tab. +1. Click the **General** tab. +1. From the **Endpoints** field, click **SAML 2.0 Identity Provider Metadata**. + +Verify the IDP metadata contains the following attributes: + +``` +xmlns:md="urn:oasis:names:tc:SAML:2.0:metadata" +xmlns:saml="urn:oasis:names:tc:SAML:2.0:assertion" +xmlns:ds="http://www.w3.org/2000/09/xmldsig#" +``` + +Some browsers, such as Firefox, may render/process the document such that the contents appear to have been modified, and some attributes appear to be missing. In this situation, use the raw response data that can be found using your browser. + +The following is an example process for Firefox, but will vary slightly for other browsers: + +1. Press **F12** to access the developer console. +1. Click the **Network** tab. +1. From the table, click the row containing `descriptor`. +1. From the details pane, click the **Response** tab. +1. Copy the raw response data. + +The XML obtained contains `EntitiesDescriptor` as the root element. Rancher expects the root element to be `EntityDescriptor` rather than `EntitiesDescriptor`. So before passing this XML to Rancher, follow these steps to adjust it: + +1. Copy all the attributes from `EntitiesDescriptor` to the `EntityDescriptor` that are not present. +1. Remove the `` tag from the beginning. +1. Remove the `` from the end of the xml. + +You are left with something similar as the example below: + +``` + +.... + +``` + +{{% /tab %}} +{{% tab "Keycloak 14+" %}} + +1. From the **Configure** section, click the **Realm Settings** tab. +1. Click the **General** tab. +1. From the **Endpoints** field, click **SAML 2.0 Identity Provider Metadata**. + +Verify the IDP metadata contains the following attributes: + +``` +xmlns:md="urn:oasis:names:tc:SAML:2.0:metadata" +xmlns:saml="urn:oasis:names:tc:SAML:2.0:assertion" +xmlns:ds="http://www.w3.org/2000/09/xmldsig#" +``` + +Some browsers, such as Firefox, may render/process the document such that the contents appear to have been modified, and some attributes appear to be missing. In this situation, use the raw response data that can be found using your browser. + +The following is an example process for Firefox, but will vary slightly for other browsers: + +1. Press **F12** to access the developer console. +1. Click the **Network** tab. +1. From the table, click the row containing `descriptor`. +1. From the details pane, click the **Response** tab. +1. Copy the raw response data. + +{{% /tab %}} +{{% /tabs %}} ## Configuring Keycloak in Rancher diff --git a/content/rancher/v2.5/en/admin-settings/authentication/ping-federate/_index.md b/content/rancher/v2.5/en/admin-settings/authentication/ping-federate/_index.md index 87abce2b877..525717bb7b9 100644 --- a/content/rancher/v2.5/en/admin-settings/authentication/ping-federate/_index.md +++ b/content/rancher/v2.5/en/admin-settings/authentication/ping-federate/_index.md @@ -30,6 +30,8 @@ Note that these URLs will not return valid data until the authentication configu 1. **Groups Field**: Make entries for managing group memberships (example: `memberOf`). + 1. **Entity ID Field** (optional): The published, protocol-dependent, unique identifier of your partner. This ID defines your organization as the entity operating the server for SAML 2.0 transactions. This ID may have been obtained out-of-band or via a SAML metadata file. + 1. **Rancher API Host**: Enter the URL for your Rancher Server. 1. **Private Key** and **Certificate**: This is a key-certificate pair to create a secure shell between Rancher and your IdP. diff --git a/content/rancher/v2.5/en/admin-settings/config-private-registry/_index.md b/content/rancher/v2.5/en/admin-settings/config-private-registry/_index.md index 378d84aea5f..d9b05763201 100644 --- a/content/rancher/v2.5/en/admin-settings/config-private-registry/_index.md +++ b/content/rancher/v2.5/en/admin-settings/config-private-registry/_index.md @@ -5,7 +5,7 @@ aliases: - /rancher/v2.x/en/admin-settings/config-private-registry/ --- -You might want to use a private Docker registry to share your custom base images within your organization. With a private registry, you can keep a private, consistent, and centralized source of truth for the Docker images that are used in your clusters. +You might want to use a private container registry to share your custom base images within your organization. With a private registry, you can keep a private, consistent, and centralized source of truth for the container images that are used in your clusters. There are two main ways to set up private registries in Rancher: by setting up the global default registry through the **Settings** tab in the global view, and by setting up a private registry in the advanced options in the cluster-level settings. The global default registry is intended to be used for air-gapped setups, for registries that do not require credentials. The cluster-level private registry is intended to be used in all setups in which the private registry requires credentials. diff --git a/content/rancher/v2.5/en/admin-settings/rbac/cluster-project-roles/_index.md b/content/rancher/v2.5/en/admin-settings/rbac/cluster-project-roles/_index.md index 6c82e42e726..4996f82ab54 100644 --- a/content/rancher/v2.5/en/admin-settings/rbac/cluster-project-roles/_index.md +++ b/content/rancher/v2.5/en/admin-settings/rbac/cluster-project-roles/_index.md @@ -86,6 +86,10 @@ _Project roles_ are roles that can be used to grant users access to a project. T These users can manage project-scoped resources like namespaces and workloads, but cannot manage other project members. + >**Note:** + > + >By default, the Rancher role of `project-member` inherits from the `Kubernetes-edit` role, and the `project-owner` role inherits from the `Kubernetes-admin` role. As such, both `project-member` and `project-owner` roles will allow for namespace management, including the ability to create and delete namespaces. + - **Read Only:** These users can view everything in the project but cannot create, update, or delete anything. @@ -94,7 +98,6 @@ _Project roles_ are roles that can be used to grant users access to a project. T > >Users assigned the `Owner` or `Member` role for a project automatically inherit the `namespace creation` role. However, this role is a [Kubernetes ClusterRole](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#role-and-clusterrole), meaning its scope extends to all projects in the cluster. Therefore, users explicitly assigned the `owner` or `member` role for a project can create namespaces in other projects they're assigned to, even with only the `Read Only` role assigned. - #### Custom Project Roles Rancher lets you assign _custom project roles_ to a standard user instead of the typical `Owner`, `Member`, or `Read Only` roles. These roles can be either a built-in custom project role or one defined by a Rancher administrator. They are convenient for defining narrow or specialized access for a standard user within a project. See the table below for a list of built-in custom project roles. diff --git a/content/rancher/v2.5/en/api/_index.md b/content/rancher/v2.5/en/api/_index.md index c6c3fdb63e3..a36f5f9fea2 100644 --- a/content/rancher/v2.5/en/api/_index.md +++ b/content/rancher/v2.5/en/api/_index.md @@ -52,3 +52,17 @@ Most collections can be sorted on the server-side by common fields using HTTP qu ## Pagination API responses are paginated with a limit of 100 resources per page by default. This can be changed with the `limit` query parameter, up to a maximum of 1000, e.g. `/v3/pods?limit=1000`. The `pagination` map in collection responses tells you whether or not you have the full result set and has a link to the next page if you do not. + +## Capturing Rancher API Calls + +You can use browser developer tools to capture how the Rancher API is called. For example, you could follow these steps to use the Chrome developer tools to get the API call for provisioning an RKE cluster: + +1. In the Rancher UI, go to **Cluster Management** and click **Create.** +1. Click one of the cluster types. This example uses Digital Ocean. +1. Fill out the form with a cluster name and node template, but don't click **Create**. +1. You will need to open the developer tools before the cluster creation to see the API call being recorded. To open the tools, right-click on the Rancher UI and click **Inspect.** +1. In the developer tools, click the **Network** tab. +1. On the **Network** tab, make sure **Fetch/XHR** is selected. +1. In the Rancher UI, click **Create**. In the developer tools, you should see a new network request with the name `cluster?_replace=true`. +1. Right-click `cluster?_replace=true` and click **Copy > Copy as cURL.** +1. Paste the result into any text editor. You will be able to see the POST request, including the URL it was sent to, all of the headers, and the full body of the request. This command can be used to create a cluster from the command line. Note: The request should be stored in a safe place because it contains credentials. \ No newline at end of file diff --git a/content/rancher/v2.5/en/backups/_index.md b/content/rancher/v2.5/en/backups/_index.md index 0f765d2d84d..01b07a0db84 100644 --- a/content/rancher/v2.5/en/backups/_index.md +++ b/content/rancher/v2.5/en/backups/_index.md @@ -63,6 +63,8 @@ The Backup and Restore custom resources can be created in the Rancher UI, or by The `rancher-backup` operator can be installed from the Rancher UI, or with the Helm CLI. In both cases, the `rancher-backup` Helm chart is installed on the Kubernetes cluster running the Rancher server. It is a cluster-admin only feature and available only for the **local** cluster. (*If you do not see `rancher-backup` in the Rancher UI, you may have selected the wrong cluster.*) +>**NOTE:** There are two known issues in Fleet that occur after performing a restoration using the backup-restore-operator: Fleet agents are inoperable and clientSecretName and helmSecretName are not included in Fleet gitrepos. Refer [here]({{}}rancher/v2.5/en/deploy-across-clusters/fleet/#troubleshooting) for workarounds. + ### Installing rancher-backup with the Rancher UI 1. In the Rancher UI's Cluster Manager, choose the cluster named **local** diff --git a/content/rancher/v2.5/en/backups/back-up-rancher/_index.md b/content/rancher/v2.5/en/backups/back-up-rancher/_index.md index 51cdf22dfc1..47f57c96e26 100644 --- a/content/rancher/v2.5/en/backups/back-up-rancher/_index.md +++ b/content/rancher/v2.5/en/backups/back-up-rancher/_index.md @@ -7,11 +7,13 @@ aliases: - /rancher/v2.x/en/backups/v2.5/back-up-rancher/ --- -In this section, you'll learn how to back up Rancher running on any Kubernetes cluster. To backup Rancher installed with Docker, refer the instructions for [single node backups]({{}}/rancher/v2.5/en/backups/v2.5/docker-installs/docker-backups) +In this section, you'll learn how to back up Rancher running on any Kubernetes cluster. To backup Rancher installed with Docker, refer to the instructions for [single node backups]({{}}/rancher/v2.5/en/backups/v2.5/docker-installs/docker-backups). The backup-restore operator needs to be installed in the local cluster, and only backs up the Rancher app. The backup and restore operations are performed only in the local Kubernetes cluster. -> When restoring a backup into a new Rancher setup, the version of the new setup should be the same as the one where the backup is made. +Note that the rancher-backup operator version 1.x.x is for Rancher v2.5.x. + +> When restoring a backup into a new Rancher setup, the version of the new setup should be the same as the one where the backup is made. The Kubernetes version should also be considered when restoring a backup, since the supported apiVersion in the cluster and in the backup file could be different. ### Prerequisites @@ -28,6 +30,8 @@ Backups are created as .tar.gz files. These files can be pushed to S3 or Minio, 1. Click **Rancher Backups.** 1. Configure the default storage location. For help, refer to the [storage configuration section.](../configuration/storage-config) +>**NOTE:** There are two known issues in Fleet that occur after performing a restoration using the backup-restore-operator: Fleet agents are inoperable and clientSecretName and helmSecretName are not included in Fleet gitrepos. Refer [here]({{}}/rancher/v2.5/en/deploy-across-clusters/fleet/#troubleshooting) for workarounds. + ### 2. Perform a Backup To perform a backup, a custom resource of type Backup must be created. diff --git a/content/rancher/v2.5/en/backups/migrating-rancher/_index.md b/content/rancher/v2.5/en/backups/migrating-rancher/_index.md index 1fc70aee590..e84fe21de1b 100644 --- a/content/rancher/v2.5/en/backups/migrating-rancher/_index.md +++ b/content/rancher/v2.5/en/backups/migrating-rancher/_index.md @@ -21,13 +21,19 @@ Rancher can be installed on any Kubernetes cluster, including hosted Kubernetes - [K3s Kubernetes installation docs]({{}}/k3s/latest/en/installation/) ### 1. Install the rancher-backup Helm chart -Install version 1.x.x of the rancher-backup chart. +Install version 1.x.x of the rancher-backup chart. The following assumes a connected environment with access to DockerHub: + ``` helm repo add rancher-charts https://charts.rancher.io helm repo update helm install rancher-backup-crd rancher-charts/rancher-backup-crd -n cattle-resources-system --create-namespace --version $CHART_VERSION helm install rancher-backup rancher-charts/rancher-backup -n cattle-resources-system --version $CHART_VERSION ``` +
+For an **air-gapped environment**, use the option below to pull the `backup-restore-operator` image from your private registry when installing the rancher-backup-crd helm chart. +``` +--set image.repository $REGISTRY/rancher/backup-restore-operator +``` ### 2. Restore from backup using a Restore custom resource @@ -70,20 +76,19 @@ spec: endpoint: s3.us-west-2.amazonaws.com ``` -> **Important:** The field `encryptionConfigSecretName` must be set only if your backup was created with encryption enabled. Provide the name of the Secret containing the encryption config file. If you only have the encryption config file, but don't have a secret created with it in this cluster, use the following steps to create the secret: -1. The encryption configuration file must be named `encryption-provider-config.yaml`, and the `--from-file` flag must be used to create this secret. So save your `EncryptionConfiguration` in a file called `encryption-provider-config.yaml` and run this command: +>**Important:** The field `encryptionConfigSecretName` must be set only if your backup was created with encryption enabled. Provide the name of the Secret containing the encryption config file. If you only have the encryption config file, but don't have a secret created with it in this cluster, use the following steps to create the secret: -``` -kubectl create secret generic encryptionconfig \ - --from-file=./encryption-provider-config.yaml \ - -n cattle-resources-system -``` - -Then apply the resource: - -``` -kubectl apply -f migrationResource.yaml -``` +1. The encryption configuration file must be named `encryption-provider-config.yaml`, and the `--from-file` flag must be used to create this secret. So save your `EncryptionConfiguration` in a file called `encryption-provider-config.yaml` and run this command: + ``` + kubectl create secret generic encryptionconfig \ + --from-file=./encryption-provider-config.yaml \ + -n cattle-resources-system + ``` + +1. Then apply the resource: + ``` + kubectl apply -f migrationResource.yaml + ``` ### 3. Install cert-manager diff --git a/content/rancher/v2.5/en/best-practices/rancher-server/deployment-types/_index.md b/content/rancher/v2.5/en/best-practices/rancher-server/deployment-types/_index.md index f32518a92f7..f7ba5e5f02e 100644 --- a/content/rancher/v2.5/en/best-practices/rancher-server/deployment-types/_index.md +++ b/content/rancher/v2.5/en/best-practices/rancher-server/deployment-types/_index.md @@ -19,7 +19,7 @@ When you set up your high-availability Rancher installation, consider the follow Don't run other workloads or microservices in the Kubernetes cluster that Rancher is installed on. ### Make sure nodes are configured correctly for Kubernetes ### -It's important to follow K8s and etcd best practices when deploying your nodes, including disabling swap, double checking you have full network connectivity between all machines in the cluster, using unique hostnames, MAC addresses, and product_uuids for every node, checking that all correct ports are opened, and deploying with ssd backed etcd. More details can be found in the [kubernetes docs](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#before-you-begin) and [etcd's performance op guide](https://github.com/etcd-io/etcd/blob/master/Documentation/op-guide/performance.md) +It's important to follow K8s and etcd best practices when deploying your nodes, including disabling swap, double checking you have full network connectivity between all machines in the cluster, using unique hostnames, MAC addresses, and product_uuids for every node, checking that all correct ports are opened, and deploying with ssd backed etcd. More details can be found in the [kubernetes docs](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#before-you-begin) and [etcd's performance op guide](https://etcd.io/docs/v3.4/op-guide/performance/). ### When using RKE: Back up the Statefile RKE keeps record of the cluster state in a file called `cluster.rkestate`. This file is important for the recovery of a cluster and/or the continued maintenance of the cluster through RKE. Because this file contains certificate material, we strongly recommend encrypting this file before backing up. After each run of `rke up` you should backup the state file. diff --git a/content/rancher/v2.5/en/best-practices/rancher-server/rancher-in-vsphere/_index.md b/content/rancher/v2.5/en/best-practices/rancher-server/rancher-in-vsphere/_index.md index 9b987726412..d3db6045085 100644 --- a/content/rancher/v2.5/en/best-practices/rancher-server/rancher-in-vsphere/_index.md +++ b/content/rancher/v2.5/en/best-practices/rancher-server/rancher-in-vsphere/_index.md @@ -53,7 +53,7 @@ https://rancher.com/docs/rancher/v2.5/en/installation/requirements/ ### Leverage VM Templates to Construct the Environment -To facilitate consistency across the deployed Virtual Machines across the environment, consider the use of "Golden Images" in the form of VM templates. Packer can be used to accomplish this, adding greater customisation options. +To facilitate the consistency of Virtual Machines deployed across the environment, consider the use of "Golden Images" in the form of VM templates. Packer can be used to accomplish this, adding greater customization options. ### Leverage DRS Anti-Affinity Rules (Where Possible) to Separate Rancher Cluster Nodes Across ESXi Hosts diff --git a/content/rancher/v2.5/en/cli/_index.md b/content/rancher/v2.5/en/cli/_index.md index 0413cdde5ed..6996b6bdf3d 100644 --- a/content/rancher/v2.5/en/cli/_index.md +++ b/content/rancher/v2.5/en/cli/_index.md @@ -1,13 +1,23 @@ --- -title: Using the Rancher Command Line Interface -description: The Rancher CLI is a unified tool that you can use to interact with Rancher. With it, you can operate Rancher using a command line interface rather than the GUI -metaTitle: "Using the Rancher Command Line Interface " -metaDescription: "The Rancher CLI is a unified tool that you can use to interact with Rancher. With it, you can operate Rancher using a command line interface rather than the GUI" +title: CLI with Rancher +description: Interact with Rancher using command line interface (CLI) tools from your workstation. weight: 21 -aliases: - - /rancher/v2.5/en/cluster-admin/cluster-access/cli --- +- [Rancher CLI](#rancher-cli) + - [Download Rancher CLI](#download-rancher-cli) + - [Requirements](#requirements) + - [CLI Authentication](#cli-authentication) + - [Project Selection](#project-selection) + - [Commands](#commands) + - [Rancher CLI Help](#rancher-cli-help) + - [Limitations](#limitations) +- [kubectl](#kubectl) + - [kubectl Utility](#kubectl-utility) + - [Authentication with kubectl and kubeconfig Tokens with TTL](#authentication-with-kubectl-and-kubeconfig-tokens-with-ttl) + +# Rancher CLI + The Rancher CLI (Command Line Interface) is a unified tool that you can use to interact with Rancher. With this tool, you can operate Rancher using a command line rather than the GUI. ### Download Rancher CLI @@ -38,11 +48,11 @@ Before you can perform any commands, you must select a Rancher project to perfor **Example: `./rancher context switch` Output** ``` User:rancher-cli-directory user$ ./rancher context switch -NUMBER CLUSTER NAME PROJECT ID PROJECT NAME -1 cluster-2 c-7q96s:p-h4tmb project-2 -2 cluster-2 c-7q96s:project-j6z6d Default -3 cluster-1 c-lchzv:p-xbpdt project-1 -4 cluster-1 c-lchzv:project-s2mch Default +NUMBER CLUSTER NAME PROJECT ID PROJECT NAME +1 cluster-2 c-7q96s:p-h4tmb project-2 +2 cluster-2 c-7q96s:project-j6z6d Default +3 cluster-1 c-lchzv:p-xbpdt project-1 +4 cluster-1 c-lchzv:project-s2mch Default Select a Project: ``` @@ -53,14 +63,16 @@ INFO[0005] Setting new context to project project-1 INFO[0005] Saving config to /Users/markbishop/.rancher/cli2.json ``` +Ensure you can run `rancher kubectl get pods` successfully. + ### Commands The following commands are available for use in Rancher CLI. | Command | Result | |---|---| -| `apps, [app]` | Performs operations on catalog applications (i.e. individual [Helm charts](https://docs.helm.sh/developing_charts/) or Rancher charts. | -| `catalog` | Performs operations on [catalogs]({{}}/rancher/v2.5/en/catalog/). | +| `apps, [app]` | Performs operations on catalog applications (i.e., individual [Helm charts](https://docs.helm.sh/developing_charts/)) or Rancher charts. | +| `catalog` | Performs operations on [catalogs]({{}}/rancher/v2.5/en/helm-charts/). | | `clusters, [cluster]` | Performs operations on your [clusters]({{}}/rancher/v2.5/en/cluster-provisioning/). | | `context` | Switches between Rancher [projects]({{}}/rancher/v2.5/en/cluster-admin/projects-and-namespaces/). For an example, see [Project Selection](#project-selection). | | `inspect [OPTIONS] [RESOURCEID RESOURCENAME]` | Displays details about [Kubernetes resources](https://kubernetes.io/docs/reference/kubectl/cheatsheet/#resource-types) or Rancher resources (i.e.: [projects]({{}}/rancher/v2.5/en/cluster-admin/projects-and-namespaces/) and [workloads]({{}}/rancher/v2.5/en/k8s-in-rancher/workloads/)). Specify resources by name or ID. | @@ -80,3 +92,38 @@ The following commands are available for use in Rancher CLI. Once logged into Rancher Server using the CLI, enter `./rancher --help` for a list of commands. All commands accept the `--help` flag, which documents each command's usage. + +### Limitations + +The Rancher CLI **cannot** be used to install [dashboard apps or Rancher feature charts](../helm-charts/). + +# kubectl + +Interact with Rancher using kubectl. + +### kubectl Utility + +Install the `kubectl` utility. See [install kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/). + +Configure kubectl by visiting your cluster in the Rancher Web UI, clicking on `Kubeconfig`, copying contents, and putting them into your `~/.kube/config` file. + +Run `kubectl cluster-info` or `kubectl get pods` successfully. + +### Authentication with kubectl and kubeconfig Tokens with TTL + +_Requirements_ + +If admins have [enforced TTL on kubeconfig tokens]({{}}/rancher/v2.5/en/api/api-tokens/#setting-ttl-on-kubeconfig-tokens), the kubeconfig file requires the [Rancher CLI](../cli) to be present in your PATH when you run `kubectl`. Otherwise, you’ll see an error like: +`Unable to connect to the server: getting credentials: exec: exec: "rancher": executable file not found in $PATH`. + +This feature enables kubectl to authenticate with the Rancher server and get a new kubeconfig token when required. The following auth providers are currently supported: + +1. Local +2. Active Directory (LDAP only) +3. FreeIPA +4. OpenLDAP +5. SAML providers: Ping, Okta, ADFS, Keycloak, Shibboleth + +When you first run kubectl, for example, `kubectl get pods`, it will ask you to pick an auth provider and log in with the Rancher server. +The kubeconfig token is cached in the path where you run kubectl under `./.cache/token`. This token is valid until [it expires](../../api/api-tokens/#setting-ttl-on-kubeconfig-tokens-period), or [gets deleted from the Rancher server](../../api/api-tokens/#deleting-tokens). +Upon expiration, the next `kubectl get pods` will ask you to log in with the Rancher server again. diff --git a/content/rancher/v2.5/en/cluster-admin/certificate-rotation/_index.md b/content/rancher/v2.5/en/cluster-admin/certificate-rotation/_index.md index b3f52ad2d3f..168a7e4094b 100644 --- a/content/rancher/v2.5/en/cluster-admin/certificate-rotation/_index.md +++ b/content/rancher/v2.5/en/cluster-admin/certificate-rotation/_index.md @@ -12,9 +12,29 @@ By default, Kubernetes clusters require certificates and Rancher launched Kubern Certificates can be rotated for the following services: - etcd -- kubelet +- kubelet (node certificate) +- kubelet (serving certificate, if [enabled]({{}}/rke/latest/en/config-options/services/#kubelet-options)) - kube-apiserver - kube-proxy - kube-scheduler - kube-controller-manager + +### Certificate Rotation + +Rancher launched Kubernetes clusters have the ability to rotate the auto-generated certificates through the UI. + +1. In the **Global** view, navigate to the cluster that you want to rotate certificates. + +2. Select **⋮ > Rotate Certificates**. + +3. Select which certificates that you want to rotate. + + * Rotate all Service certificates (keep the same CA) + * Rotate an individual service and choose one of the services from the drop-down menu + +4. Click **Save**. + +**Results:** The selected certificates will be rotated and the related services will be restarted to start using the new certificate. + +> **Note:** Even though the RKE CLI can use custom certificates for the Kubernetes cluster components, Rancher currently doesn't allow the ability to upload these in Rancher launched Kubernetes clusters. diff --git a/content/rancher/v2.5/en/cluster-admin/editing-clusters/eks-config-reference/_index.md b/content/rancher/v2.5/en/cluster-admin/editing-clusters/eks-config-reference/_index.md index bb4e31086c1..0ea3ab8ad80 100644 --- a/content/rancher/v2.5/en/cluster-admin/editing-clusters/eks-config-reference/_index.md +++ b/content/rancher/v2.5/en/cluster-admin/editing-clusters/eks-config-reference/_index.md @@ -16,7 +16,7 @@ Complete each drop-down and field using the information obtained for your IAM po | Setting | Description | | ---------- | -------------------------------------------------------------------------------------------------------------------- | | Region | From the drop-down choose the geographical region in which to build your cluster. | -| Cloud Credentials | Select the cloud credentials that you created for your IAM policy. For more information on creating cloud credentials in Rancher, refer to [this page.]({{}}/rancher/v2.x/en/user-settings/cloud-credentials/) | +| Cloud Credentials | Select the cloud credentials that you created for your IAM policy. For more information on creating cloud credentials in Rancher, refer to [this page.]({{}}/rancher/v2.5/en/user-settings/cloud-credentials/) | ### Service Role @@ -186,7 +186,7 @@ Complete each drop-down and field using the information obtained for your IAM po | Setting | Description | | ---------- | -------------------------------------------------------------------------------------------------------------------- | | Region | From the drop-down choose the geographical region in which to build your cluster. | -| Cloud Credentials | Select the cloud credentials that you created for your IAM policy. For more information on creating cloud credentials in Rancher, refer to [this page.]({{}}/rancher/v2.x/en/user-settings/cloud-credentials/) | +| Cloud Credentials | Select the cloud credentials that you created for your IAM policy. For more information on creating cloud credentials in Rancher, refer to [this page.]({{}}/rancher/v2.5/en/user-settings/cloud-credentials/) | ### Service Role diff --git a/content/rancher/v2.5/en/cluster-admin/editing-clusters/gke-config-reference/private-clusters/_index.md b/content/rancher/v2.5/en/cluster-admin/editing-clusters/gke-config-reference/private-clusters/_index.md index d66fdb087bd..9de7dbf3c1d 100644 --- a/content/rancher/v2.5/en/cluster-admin/editing-clusters/gke-config-reference/private-clusters/_index.md +++ b/content/rancher/v2.5/en/cluster-admin/editing-clusters/gke-config-reference/private-clusters/_index.md @@ -25,7 +25,7 @@ If restricting outgoing internet access is not a concern for your organization, >**Note** >This scenario is not officially supported, but is described for cases in which using the Cloud NAT service is not sufficient. -If restricting both incoming and outgoing traffic to nodes is a requirement, follow the air-gapped installation instructions to set up a private container image [registry](https://rancher.com/docs/rancher/v2.x/en/installation/other-installation-methods/air-gap/) on the VPC where the cluster is going to be, allowing the cluster nodes to access and download the images they need to run the cluster agent. If the control plane endpoint is also private, Rancher will need [direct access](#direct-access) to it. +If restricting both incoming and outgoing traffic to nodes is a requirement, follow the air-gapped installation instructions to set up a private container image [registry](https://rancher.com/docs/rancher/v2.5/en/installation/other-installation-methods/air-gap/) on the VPC where the cluster is going to be, allowing the cluster nodes to access and download the images they need to run the cluster agent. If the control plane endpoint is also private, Rancher will need [direct access](#direct-access) to it. ### Private Control Plane Endpoint diff --git a/content/rancher/v2.5/en/cluster-admin/nodes/_index.md b/content/rancher/v2.5/en/cluster-admin/nodes/_index.md index 514ebc67a18..af48e6fa950 100644 --- a/content/rancher/v2.5/en/cluster-admin/nodes/_index.md +++ b/content/rancher/v2.5/en/cluster-admin/nodes/_index.md @@ -42,11 +42,11 @@ The following table lists which node options are available for each type of clus | [Download Keys](#ssh-into-a-node-hosted-by-an-infrastructure-provider) | ✓ | | | | | Download SSH key in order to SSH into the node. | | [Node Scaling](#scaling-nodes) | ✓ | | | ✓ | | Scale the number of nodes in the node pool up or down. | -[1]: {{}}/rancher/v2.x/en/cluster-provisioning/rke-clusters/node-pools/ -[2]: {{}}/rancher/v2.x/en/cluster-provisioning/rke-clusters/custom-nodes/ -[3]: {{}}/rancher/v2.x/en/cluster-provisioning/hosted-kubernetes-clusters/ -[4]: {{}}/rancher/v2.x/en/cluster-provisioning/registered-clusters/ -[5]: {{}}/rancher/v2.x/en/cluster-provisioning/registered-clusters/ +[1]: {{}}/rancher/v2.5/en/cluster-provisioning/rke-clusters/node-pools/ +[2]: {{}}/rancher/v2.5/en/cluster-provisioning/rke-clusters/custom-nodes/ +[3]: {{}}/rancher/v2.5/en/cluster-provisioning/hosted-kubernetes-clusters/ +[4]: {{}}/rancher/v2.5/en/cluster-provisioning/registered-clusters/ +[5]: {{}}/rancher/v2.5/en/cluster-provisioning/registered-clusters/ \* Delete option accessible via View API diff --git a/content/rancher/v2.5/en/cluster-admin/projects-and-namespaces/_index.md b/content/rancher/v2.5/en/cluster-admin/projects-and-namespaces/_index.md index 86c6574c224..493331bc93c 100644 --- a/content/rancher/v2.5/en/cluster-admin/projects-and-namespaces/_index.md +++ b/content/rancher/v2.5/en/cluster-admin/projects-and-namespaces/_index.md @@ -156,6 +156,9 @@ By default, your user is added as the project `Owner`. >**Notes on Permissions:** > >- Users assigned the `Owner` or `Member` role for a project automatically inherit the `namespace creation` role. However, this role is a [Kubernetes ClusterRole](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#role-and-clusterrole), meaning its scope extends to all projects in the cluster. Therefore, users explicitly assigned the `Owner` or `Member` role for a project can create namespaces in other projects they're assigned to, even with only the `Read Only` role assigned. +> +>- By default, the Rancher role of `project-member` inherits from the `Kubernetes-edit` role, and the `project-owner` role inherits from the `Kubernetes-admin` role. As such, both `project-member` and `project-owner` roles will allow for namespace management, including the ability to create and delete namespaces. +> >- Choose `Custom` to create a custom role on the fly: [Custom Project Roles]({{}}/rancher/v2.5/en/admin-settings/rbac/cluster-project-roles/#custom-project-roles). To add members: diff --git a/content/rancher/v2.5/en/cluster-admin/volumes-and-storage/iscsi-volumes/_index.md b/content/rancher/v2.5/en/cluster-admin/volumes-and-storage/iscsi-volumes/_index.md index 1a30d52540d..deaaaac4fc5 100644 --- a/content/rancher/v2.5/en/cluster-admin/volumes-and-storage/iscsi-volumes/_index.md +++ b/content/rancher/v2.5/en/cluster-admin/volumes-and-storage/iscsi-volumes/_index.md @@ -19,9 +19,12 @@ If you encounter this issue, you can work around it by installing the initiator After installing the initiator tool on your nodes, edit the YAML for your cluster, editing the kubelet configuration to mount the iSCSI binary and configuration, as shown in the sample below. ->**Note:** +>**Notes:** +> +>- Before updating your Kubernetes YAML to mount the iSCSI binary and configuration, make sure either the `open-iscsi` (deb) or `iscsi-initiator-utils` (yum) package is installed on your cluster nodes. If this package isn't installed _before_ the bind mounts are created in your Kubernetes YAML, Docker will automatically create the directories and files on each node and will not allow the package install to succeed.
+>
> ->Before updating your Kubernetes YAML to mount the iSCSI binary and configuration, make sure either the `open-iscsi` (deb) or `iscsi-initiator-utils` (yum) package is installed on your cluster nodes. If this package isn't installed _before_ the bind mounts are created in your Kubernetes YAML, Docker will automatically create the directories and files on each node and will not allow the package install to succeed. +>- The example YAML below does not apply to K3s, but only to RKE clusters. Since the K3s kubelet does not run in a container, adding extra binds is not necessary. However, all iSCSI tools must still be installed on your K3s nodes. ``` services: diff --git a/content/rancher/v2.5/en/cluster-provisioning/hosted-kubernetes-clusters/eks/permissions/_index.md b/content/rancher/v2.5/en/cluster-provisioning/hosted-kubernetes-clusters/eks/permissions/_index.md index 0567e110d02..8f4539e1c6f 100644 --- a/content/rancher/v2.5/en/cluster-provisioning/hosted-kubernetes-clusters/eks/permissions/_index.md +++ b/content/rancher/v2.5/en/cluster-provisioning/hosted-kubernetes-clusters/eks/permissions/_index.md @@ -24,6 +24,7 @@ Resource targeting uses `*` as the ARN of many of the resources created cannot b "ec2:RunInstances", "ec2:RevokeSecurityGroupIngress", "ec2:RevokeSecurityGroupEgress", + "ec2:DescribeRegions", "ec2:DescribeVpcs", "ec2:DescribeTags", "ec2:DescribeSubnets", @@ -123,31 +124,6 @@ Resource targeting uses `*` as the ARN of many of the resources created cannot b ### Service Role Permissions -Rancher will create a service role with the following trust policy: - -```json -{ - "Version": "2012-10-17", - "Statement": [ - { - "Action": "sts:AssumeRole", - "Principal": { - "Service": "eks.amazonaws.com" - }, - "Effect": "Allow", - "Sid": "" - } - ] -} -``` - -This role will also have two role policy attachments with the following policies ARNs: - -``` -arn:aws:iam::aws:policy/AmazonEKSClusterPolicy -arn:aws:iam::aws:policy/AmazonEKSServicePolicy -``` - Permissions required for Rancher to create service role on users behalf during the EKS cluster creation process. ```json @@ -182,36 +158,66 @@ Permissions required for Rancher to create service role on users behalf during t } ``` +When an EKS cluster is created, Rancher will create a service role with the following trust policy: + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Action": "sts:AssumeRole", + "Principal": { + "Service": "eks.amazonaws.com" + }, + "Effect": "Allow", + "Sid": "" + } + ] +} +``` + +This role will also have two role policy attachments with the following policies ARNs: + +``` +arn:aws:iam::aws:policy/AmazonEKSClusterPolicy +arn:aws:iam::aws:policy/AmazonEKSServicePolicy +``` + ### VPC Permissions Permissions required for Rancher to create VPC and associated resources. ```json { - "Sid": "VPCPermissions", - "Effect": "Allow", - "Action": [ - "ec2:ReplaceRoute", - "ec2:ModifyVpcAttribute", - "ec2:ModifySubnetAttribute", - "ec2:DisassociateRouteTable", - "ec2:DetachInternetGateway", - "ec2:DescribeVpcs", - "ec2:DeleteVpc", - "ec2:DeleteTags", - "ec2:DeleteSubnet", - "ec2:DeleteRouteTable", - "ec2:DeleteRoute", - "ec2:DeleteInternetGateway", - "ec2:CreateVpc", - "ec2:CreateSubnet", - "ec2:CreateSecurityGroup", - "ec2:CreateRouteTable", - "ec2:CreateRoute", - "ec2:CreateInternetGateway", - "ec2:AttachInternetGateway", - "ec2:AssociateRouteTable" - ], - "Resource": "*" + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "VPCPermissions", + "Effect": "Allow", + "Action": [ + "ec2:ReplaceRoute", + "ec2:ModifyVpcAttribute", + "ec2:ModifySubnetAttribute", + "ec2:DisassociateRouteTable", + "ec2:DetachInternetGateway", + "ec2:DescribeVpcs", + "ec2:DeleteVpc", + "ec2:DeleteTags", + "ec2:DeleteSubnet", + "ec2:DeleteRouteTable", + "ec2:DeleteRoute", + "ec2:DeleteInternetGateway", + "ec2:CreateVpc", + "ec2:CreateSubnet", + "ec2:CreateSecurityGroup", + "ec2:CreateRouteTable", + "ec2:CreateRoute", + "ec2:CreateInternetGateway", + "ec2:AttachInternetGateway", + "ec2:AssociateRouteTable" + ], + "Resource": "*" + } + ] } -``` \ No newline at end of file +``` diff --git a/content/rancher/v2.5/en/cluster-provisioning/node-requirements/_index.md b/content/rancher/v2.5/en/cluster-provisioning/node-requirements/_index.md index 54e83ca885d..3daf9c0cb8f 100644 --- a/content/rancher/v2.5/en/cluster-provisioning/node-requirements/_index.md +++ b/content/rancher/v2.5/en/cluster-provisioning/node-requirements/_index.md @@ -32,6 +32,13 @@ For information on how to install Docker, refer to the official [Docker document Some distributions of Linux derived from RHEL, including Oracle Linux, may have default firewall rules that block communication with Helm. We recommend disabling firewalld. For Kubernetes 1.19, firewalld must be turned off. +>**Note:** In RHEL 8.4, two extra services are included on the NetworkManager: `nm-cloud-setup.service` and `nm-cloud-setup.timer`. These services add a routing table that interferes with the CNI plugin's configuration. If these services are enabled, you must disable them using the command below, and then reboot the node to restore connectivity: +> +> ``` + systemctl disable nm-cloud-setup.service nm-cloud-setup.timer + reboot + ``` + ### SUSE Linux Nodes SUSE Linux may have a firewall that blocks all ports by default. In that situation, follow [these steps]({{}}/rancher/v2.5/en/installation/requirements/ports/#opening-suse-linux-ports) to open the ports needed for adding a host to a custom cluster. diff --git a/content/rancher/v2.5/en/cluster-provisioning/registered-clusters/_index.md b/content/rancher/v2.5/en/cluster-provisioning/registered-clusters/_index.md index 8791b91df96..0e1ee65c004 100644 --- a/content/rancher/v2.5/en/cluster-provisioning/registered-clusters/_index.md +++ b/content/rancher/v2.5/en/cluster-provisioning/registered-clusters/_index.md @@ -20,6 +20,17 @@ The control that Rancher has to manage a registered cluster depends on the type # Prerequisites +{{% tabs %}} +{{% tab "v2.5.9+" %}} + +### Kubernetes Node Roles + +Registered RKE Kubernetes clusters must have all three node roles - etcd, controlplane and worker. A cluster with only controlplane components cannot be registered in Rancher. + +For more information on RKE node roles, see the [best practices.]({{}}/rancher/v2.5/en/cluster-provisioning/production/#cluster-architecture) + +### Permissions + If your existing Kubernetes cluster already has a `cluster-admin` role defined, you must have this `cluster-admin` privilege to register the cluster in Rancher. In order to apply the privilege, you need to run: @@ -36,6 +47,38 @@ By default, GKE users are not given this privilege, so you will need to run the If you are registering a K3s cluster, make sure the `cluster.yml` is readable. It is protected by default. For details, refer to [Configuring a K3s cluster to enable importation to Rancher.](#configuring-a-k3s-cluster-to-enable-registration-in-rancher) +### EKS Clusters + +EKS clusters must have at least one managed node group to be imported into Rancher or provisioned from Rancher successfully. + +{{% /tab %}} +{{% tab "Rancher before v2.5.9" %}} + +### Permissions + +If your existing Kubernetes cluster already has a `cluster-admin` role defined, you must have this `cluster-admin` privilege to register the cluster in Rancher. + +In order to apply the privilege, you need to run: + +```plain +kubectl create clusterrolebinding cluster-admin-binding \ + --clusterrole cluster-admin \ + --user [USER_ACCOUNT] +``` + +before running the `kubectl` command to register the cluster. + +By default, GKE users are not given this privilege, so you will need to run the command before registering GKE clusters. To learn more about role-based access control for GKE, please click [here](https://cloud.google.com/kubernetes-engine/docs/how-to/role-based-access-control). + +If you are registering a K3s cluster, make sure the `cluster.yml` is readable. It is protected by default. For details, refer to [Configuring a K3s cluster to enable importation to Rancher.](#configuring-a-k3s-cluster-to-enable-registration-in-rancher) + +### EKS Clusters + +EKS clusters must have at least one managed node group to be imported into Rancher or provisioned from Rancher successfully. + +{{% /tab %}} +{{% /tabs %}} + # Registering a Cluster 1. From the **Clusters** page, click **Add Cluster**. @@ -76,6 +119,34 @@ The option can also be specified using the environment variable `K3S_KUBECONFIG_ $ curl -sfL https://get.k3s.io | K3S_KUBECONFIG_MODE="644" sh -s - ``` +### Configuring an Imported EKS Cluster with Terraform + +You should define **only** the minimum fields that Rancher requires when importing an EKS cluster with Terraform. This is important as Rancher will overwrite what was in the EKS cluster with any config that the user has provided. + +>**Warning:** Even a small difference between the current EKS cluster and a user-provided config could have unexpected results. + +The minimum config fields required by Rancher to import EKS clusters with Terraform using `eks_config_v2` are as follows: + +- cloud_credential_id +- name +- region +- imported (this field should always be set to `true` for imported clusters) + +Example YAML configuration for imported EKS clusters: + +``` +resource "rancher2_cluster" "my-eks-to-import" { + name = "my-eks-to-import" + description = "Terraform EKS Cluster" + eks_config_v2 { + cloud_credential_id = rancher2_cloud_credential.aws.id + name = var.aws_eks_name + region = var.aws_region + imported = true + } +} +``` + # Management Capabilities for Registered Clusters The control that Rancher has to manage a registered cluster depends on the type of cluster. diff --git a/content/rancher/v2.5/en/cluster-provisioning/rke-clusters/cloud-providers/_index.md b/content/rancher/v2.5/en/cluster-provisioning/rke-clusters/cloud-providers/_index.md index 82f660865e9..2679bbe634b 100644 --- a/content/rancher/v2.5/en/cluster-provisioning/rke-clusters/cloud-providers/_index.md +++ b/content/rancher/v2.5/en/cluster-provisioning/rke-clusters/cloud-providers/_index.md @@ -6,7 +6,7 @@ aliases: - /rancher/v2.5/en/cluster-provisioning/rke-clusters/options/cloud-providers - /rancher/v2.x/en/cluster-provisioning/rke-clusters/cloud-providers/ --- -A _cloud provider_ is a module in Kubernetes that provides an interface for managing nodes, load balancers, and networking routes. For more information, refer to the [official Kubernetes documentation on cloud providers.](https://kubernetes.io/docs/concepts/cluster-administration/cloud-providers/) +A _cloud provider_ is a module in Kubernetes that provides an interface for managing nodes, load balancers, and networking routes. When a cloud provider is set up in Rancher, the Rancher server can automatically provision new nodes, load balancers or persistent storage devices when launching Kubernetes definitions, if the cloud provider you're using supports such automation. @@ -39,9 +39,9 @@ For details on enabling the vSphere cloud provider, refer to [this page.](./vsph ### Setting up a Custom Cloud Provider -The `Custom` cloud provider is available if you want to configure any [Kubernetes cloud provider](https://kubernetes.io/docs/concepts/cluster-administration/cloud-providers/). +The `Custom` cloud provider is available if you want to configure any Kubernetes cloud provider. -For the custom cloud provider option, you can refer to the [RKE docs]({{}}/rke/latest/en/config-options/cloud-providers/) on how to edit the yaml file for your specific cloud provider. There are specific cloud providers that have more detailed configuration : +For the custom cloud provider option, you can refer to the [RKE docs]({{}}/rke/latest/en/config-options/cloud-providers/) on how to edit the yaml file for your specific cloud provider. There are specific cloud providers that have more detailed configuration: * [vSphere]({{}}/rke/latest/en/config-options/cloud-providers/vsphere/) * [OpenStack]({{}}/rke/latest/en/config-options/cloud-providers/openstack/) diff --git a/content/rancher/v2.5/en/cluster-provisioning/rke-clusters/cloud-providers/vsphere/out-of-tree/_index.md b/content/rancher/v2.5/en/cluster-provisioning/rke-clusters/cloud-providers/vsphere/out-of-tree/_index.md index 33c3a39e9d8..545d87e0f49 100644 --- a/content/rancher/v2.5/en/cluster-provisioning/rke-clusters/cloud-providers/vsphere/out-of-tree/_index.md +++ b/content/rancher/v2.5/en/cluster-provisioning/rke-clusters/cloud-providers/vsphere/out-of-tree/_index.md @@ -13,7 +13,10 @@ This page covers how to install the Cloud Provider Interface (CPI) and Cloud Sto # Prerequisites -The vSphere version must be 7.0u1 or higher. +The vSphere versions supported: + +* 6.7u3 +* 7.0u1 or higher. The Kubernetes version must be 1.19 or higher. diff --git a/content/rancher/v2.5/en/cluster-provisioning/rke-clusters/options/_index.md b/content/rancher/v2.5/en/cluster-provisioning/rke-clusters/options/_index.md index 662af99a33d..63b3bfddcbd 100644 --- a/content/rancher/v2.5/en/cluster-provisioning/rke-clusters/options/_index.md +++ b/content/rancher/v2.5/en/cluster-provisioning/rke-clusters/options/_index.md @@ -158,7 +158,7 @@ Option to enable or disable [recurring etcd snapshots]({{}}/rke/latest/ _Available as of v2.5.6_ -Option to set environment variables for [rancher agents]({{}}/rancher/v2.x/en/cluster-provisioning/rke-clusters/rancher-agents/). The environment variables can be set using key value pairs. If rancher agent requires use of proxy to communicate with Rancher server, `HTTP_PROXY`, `HTTPS_PROXY` and `NO_PROXY` environment variables can be set using agent environment variables. +Option to set environment variables for [rancher agents]({{}}/rancher/v2.5/en/cluster-provisioning/rke-clusters/rancher-agents/). The environment variables can be set using key value pairs. If rancher agent requires use of proxy to communicate with Rancher server, `HTTP_PROXY`, `HTTPS_PROXY` and `NO_PROXY` environment variables can be set using agent environment variables. # Cluster Config File diff --git a/content/rancher/v2.5/en/cluster-provisioning/rke-clusters/windows-clusters/_index.md b/content/rancher/v2.5/en/cluster-provisioning/rke-clusters/windows-clusters/_index.md index 37ff393eded..16bf34656c1 100644 --- a/content/rancher/v2.5/en/cluster-provisioning/rke-clusters/windows-clusters/_index.md +++ b/content/rancher/v2.5/en/cluster-provisioning/rke-clusters/windows-clusters/_index.md @@ -85,7 +85,7 @@ If you are configuring DHCP options sets for an AWS virtual private cloud, note > Some Linux operating systems accept multiple domain names separated by spaces. However, other Linux operating systems and Windows treat the value as a single domain, which results in unexpected behavior. If your DHCP options set is associated with a VPC that has instances with multiple operating systems, specify only one domain name. -#### Rancher on vSphere with ESXi 6.7u2 and above +### Rancher on vSphere with ESXi 6.7u2 and above If you are using Rancher on VMware vSphere with ESXi 6.7u2 or later with Red Hat Enterprise Linux 8.3, CentOS 8.3, or SUSE Enterprise Linux 15 SP2 or later, it is necessary to disable the `vmxnet3` virtual network adapter hardware offloading feature. Failure to do so will result in all network connections between pods on different cluster nodes to fail with timeout errors. All connections from Windows pods to critical services running on Linux nodes, such as CoreDNS, will fail as well. It is also possible that external connections may fail. This issue is the result of Linux distributions enabling the hardware offloading feature in `vmxnet3` and a bug in the `vmxnet3` hardware offloading feature that results in the discarding of packets for guest overlay traffic. To address this issue, it is necessary disable the `vmxnet3` hardware offloading feature. This setting does not survive reboot, so it is necessary to disable on every boot. The recommended course of action is to create a systemd unit file at `/etc/systemd/system/disable_hw_offloading.service`, which disables the `vmxnet3` hardware offloading feature on boot. A sample systemd unit file which disables the `vmxnet3` hardware offloading feature is as follows. Note that `` must be customized to the host `vmxnet3` network interface, e.g., `ens192`: diff --git a/content/rancher/v2.5/en/deploy-across-clusters/fleet/_index.md b/content/rancher/v2.5/en/deploy-across-clusters/fleet/_index.md index abea9bad31e..9774589340a 100644 --- a/content/rancher/v2.5/en/deploy-across-clusters/fleet/_index.md +++ b/content/rancher/v2.5/en/deploy-across-clusters/fleet/_index.md @@ -16,6 +16,8 @@ Fleet is a separate project from Rancher, and can be installed on any Kubernetes - [Windows Support](#windows-support) - [GitHub Repository](#github-repository) - [Using Fleet Behind a Proxy](#using-fleet-behind-a-proxy) +- [Helm Chart Dependencies](#helm-chart-dependencies) +- [Troubleshooting](#troubleshooting) - [Documentation](#documentation) # Architecture @@ -24,7 +26,24 @@ For information about how Fleet works, see [this page.](./architecture) # Accessing Fleet in the Rancher UI -Fleet comes preinstalled in Rancher v2.5. To access it, go to the **Cluster Explorer** in the Rancher UI. In the top left dropdown menu, click **Cluster Explorer > Continuous Delivery.** On this page, you can edit Kubernetes resources and cluster groups managed by Fleet. +Fleet comes preinstalled in Rancher v2.5. Users can leverage continuous delivery to deploy their applications to the Kubernetes clusters in the git repository without any manual operation by following **gitops** practice. For additional information on Continuous Delivery and other Fleet troubleshooting tips, refer [here](https://fleet.rancher.io/troubleshooting/). + +Follow the steps below to access Continuous Delivery in the Rancher UI: + +1. Click **Cluster Explorer** in the Rancher UI. + +1. In the top left dropdown menu, click **Cluster Explorer > Continuous Delivery.** + +1. Select your namespace at the top of the menu, noting the following: + - By default,`fleet-default` is selected which includes all downstream clusters that are registered through Rancher. + - You may switch to `fleet-local`, which only contains the `local` cluster, or you may create your own workspace to which you may assign and move clusters. + - You can then manage clusters by clicking on **Clusters** on the left navigation bar. + +1. Click on **Gitrepos** on the left navigation bar to deploy the gitrepo into your clusters in the current workspace. + +1. Select your [git repository](https://fleet.rancher.io/gitrepo-add/) and [target clusters/cluster group](https://fleet.rancher.io/gitrepo-structure/). You can also create the cluster group in the UI by clicking on **Cluster Groups** from the left navigation bar. + +1. Once the gitrepo is deployed, you can monitor the application through the Rancher UI. # Windows Support @@ -44,6 +63,31 @@ _Available as of v2.5.8_ For details on using Fleet behind a proxy, see [this page.](./proxy) +# Helm Chart Dependencies + +In order for Helm charts with dependencies to deploy successfully, you must run a manual command (as listed below), as it is up to the user to fulfill the dependency list. If you do not do this and proceed to clone your repository and run `helm install`, your installation will fail because the dependencies will be missing. + +The Helm chart in the git repository must include its dependencies in the charts subdirectory. You must either manually run `helm dependencies update $chart` OR run `helm dependencies build $chart` locally, then commit the complete charts directory to your git repository. Note that you will update your commands with the applicable parameters. + +# Troubleshooting +--- +* **Known Issue:** Fleet becomes inoperable after a restore using the [backup-restore-operator]({{}}/rancher/v2.5/en/backups/back-up-rancher/#1-install-the-rancher-backup-operator). We will update the community once a permanent solution is in place. + +* **Temporary Workaround:**
+ 1. Find the two service account tokens listed in the fleet-controller and the fleet-controller-bootstrap service accounts. These are under the fleet-system namespace of the local cluster.
+ 2. Remove the non-existent token secret. Doing so allows for only one entry to be present for the service account token secret that actually exists.
+ 3. Delete the fleet-controller Pod in the fleet-system namespace to reschedule.
+ 4. After the service account token issue is resolved, you can force redeployment of the fleet-agents. In the Rancher UI, go to **☰ > Cluster Management**, click on **Clusters** page, then click **Force Update**.
+ 5. If the fleet-agent bundles remain in a `Modified` state after Step 4, update the field `spec.forceSyncGeneration` for the fleet-agent bundle to force re-creation. + +--- +* **Known Issue:** clientSecretName and helmSecretName secrets for Fleet gitrepos are not included in the backup nor restore created by the [backup-restore-operator]({{}}/rancher/v2.5/en/backups/back-up-rancher/#1-install-the-rancher-backup-operator). We will update the community once a permanent solution is in place. + +* **Temporary Workaround:**
+By default, user-defined secrets are not backed up in Fleet. It is necessary to recreate secrets if performing a disaster recovery restore or migration of Rancher into a fresh cluster. To modify resourceSet to include extra resources you want to backup, refer to docs [here](https://github.com/rancher/backup-restore-operator#user-flow). + +--- + # Documentation The Fleet documentation is at [https://fleet.rancher.io/.](https://fleet.rancher.io/) diff --git a/content/rancher/v2.5/en/faq/removing-rancher/_index.md b/content/rancher/v2.5/en/faq/removing-rancher/_index.md index 49c1acde9bc..e05744e7400 100644 --- a/content/rancher/v2.5/en/faq/removing-rancher/_index.md +++ b/content/rancher/v2.5/en/faq/removing-rancher/_index.md @@ -27,7 +27,7 @@ The capability to access a downstream cluster without Rancher depends on the typ - **Registered clusters:** The cluster will be unaffected and you can access the cluster using the same methods that you did before the cluster was registered into Rancher. - **Hosted Kubernetes clusters:** If you created the cluster in a cloud-hosted Kubernetes provider such as EKS, GKE, or AKS, you can continue to manage the cluster using your provider's cloud credentials. -- **RKE clusters:** To access an [RKE cluster,]({{}}/rancher/v2.5/en/cluster-provisioning/rke-clusters/) the cluster must have the [authorized cluster endpoint]({{}}/rancher/v2.5/en/overview/architecture/#4-authorized-cluster-endpoint) enabled, and you must have already downloaded the cluster's kubeconfig file from the Rancher UI. (The authorized cluster endpoint is enabled by default for RKE clusters.) With this endpoint, you can access your cluster with kubectl directly instead of communicating through the Rancher server's [authentication proxy.]({{}}/rancher/v2.5/en/overview/architecture/#1-the-authentication-proxy) For instructions on how to configure kubectl to use the authorized cluster endpoint, refer to the section about directly accessing clusters with [kubectl and the kubeconfig file.]({{}}/rancher/v2.5/en/cluster-admin/cluster-access/kubectl/#authenticating-directly-with-a-downstream-cluster) These clusters will use a snapshot of the authentication as it was configured when Rancher was removed. +- **RKE clusters:** Please note that you will no longer be able to manage the individual Kubernetes components or perform any upgrades on them after the deletion of the Rancher server. However, you can still access the cluster to manage your workloads. To access an [RKE cluster,]({{}}/rancher/v2.5/en/cluster-provisioning/rke-clusters/) the cluster must have the [authorized cluster endpoint]({{}}/rancher/v2.5/en/overview/architecture/#4-authorized-cluster-endpoint) enabled, and you must have already downloaded the cluster's kubeconfig file from the Rancher UI. (The authorized cluster endpoint is enabled by default for RKE clusters.) With this endpoint, you can access your cluster with kubectl directly instead of communicating through the Rancher server's [authentication proxy.]({{}}/rancher/v2.5/en/overview/architecture/#1-the-authentication-proxy) For instructions on how to configure kubectl to use the authorized cluster endpoint, refer to the section about directly accessing clusters with [kubectl and the kubeconfig file.]({{}}/rancher/v2.5/en/cluster-admin/cluster-access/kubectl/#authenticating-directly-with-a-downstream-cluster) These clusters will use a snapshot of the authentication as it was configured when Rancher was removed. ### What if I don't want Rancher anymore? diff --git a/content/rancher/v2.5/en/faq/technical/_index.md b/content/rancher/v2.5/en/faq/technical/_index.md index f338f2a0180..9ea91ea6d92 100644 --- a/content/rancher/v2.5/en/faq/technical/_index.md +++ b/content/rancher/v2.5/en/faq/technical/_index.md @@ -173,3 +173,8 @@ In Kubernetes v1.13, the `TaintBasedEvictions` feature is enabled by default. Se ### Can I use keyboard shortcuts in the UI? Yes, most parts of the UI can be reached using keyboard shortcuts. For an overview of the available shortcuts, press `?` anywhere in the UI. + + +### What does `Unknown schema for type:` errors followed by something like `catalog.cattle.io.operation` mean when trying to modify an App? + +This error occurs when Kubernetes can not find the CRD mentioned. The vast majority of the time these are a result of missing RBAC permissions. Try with an admin user and if this works, add permissions for the resource mentioned by the error (ie. `Get`, `List`, `Patch` as needed). diff --git a/content/rancher/v2.5/en/helm-charts/_index.md b/content/rancher/v2.5/en/helm-charts/_index.md index e5e6ba7853e..b74682a0c62 100644 --- a/content/rancher/v2.5/en/helm-charts/_index.md +++ b/content/rancher/v2.5/en/helm-charts/_index.md @@ -50,6 +50,39 @@ From the left sidebar select _"Repositories"_. These items represent helm repositories, and can be either traditional helm endpoints which have an index.yaml, or git repositories which will be cloned and can point to a specific branch. In order to use custom charts, simply add your repository here and they will become available in the Charts tab under the name of the repository. +To add a private CA for Helm Chart repositories: + +- **HTTP-based chart repositories**: You must add a base64 encoded copy of the CA certificate in DER format to the spec.caBundle field of the chart repo, such as `openssl x509 -outform der -in ca.pem | base64 -w0`. Click **Edit YAML** for the chart repo and set, as in the following example:
+ ``` + [...] + spec: + caBundle: + MIIFXzCCA0egAwIBAgIUWNy8WrvSkgNzV0zdWRP79j9cVcEwDQYJKoZIhvcNAQELBQAwPzELMAkGA1UEBhMCVVMxCzAJBgNVBAgMAkNBMRQwEgYDVQQKDAtNeU9yZywgSW5jLjENMAsGA1UEAwwEcm9vdDAeFw0yMTEyMTQwODMyMTdaFw0yNDEwMDMwODMyMT + ... + nDxZ/tNXt/WPJr/PgEB3hQdInDWYMg7vGO0Oz00G5kWg0sJ0ZTSoA10ZwdjIdGEeKlj1NlPyAqpQ+uDnmx6DW+zqfYtLnc/g6GuLLVPamraqN+gyU8CHwAWPNjZonFN9Vpg0PIk1I2zuOc4EHifoTAXSpnjfzfyAxCaZsnTptimlPFJJqAMj+FfDArGmr4= + [...] + ``` + +- **Git-based chart repositories**: It is not currently possible to add a private CA. For git-based chart repositories with a certificate signed by a private CA, you must disable TLS verification. Click **Edit YAML** for the chart repo, and add the key/value pair as follows: + ``` + [...] + spec: + insecureSkipTLSVerify: true + [...] + ``` + +> **Note:** Helm chart repositories with authentication +> +> As of Rancher v2.5.12, a new value `disableSameOriginCheck` has been added to the Repo.Spec. This allows users to bypass the same origin checks, sending the repository Authentication information as a Basic Auth Header with all API calls. This is not recommended but can be used as a temporary solution in cases of non-standard Helm chart repositories such as those that have redirects to a different origin URL. +> +> To use this feature for an existing Helm chart repository, click ⋮ > Edit YAML. On the `spec` portion of the YAML file, add `disableSameOriginCheck` and set it to `true`. +> +> ```yaml +[...] +spec: + disableSameOriginCheck: true +[...] +``` ### Helm Compatibility diff --git a/content/rancher/v2.5/en/installation/_index.md b/content/rancher/v2.5/en/installation/_index.md index 409e36d4d8e..9f49f7a1b73 100644 --- a/content/rancher/v2.5/en/installation/_index.md +++ b/content/rancher/v2.5/en/installation/_index.md @@ -17,7 +17,7 @@ In this section, - **RKE (Rancher Kubernetes Engine)** is a certified Kubernetes distribution and CLI/library which creates and manages a Kubernetes cluster. - **K3s (Lightweight Kubernetes)** is also a fully compliant Kubernetes distribution. It is newer than RKE, easier to use, and more lightweight, with a binary size of less than 100 MB. - **RKE2** is a fully conformant Kubernetes distribution that focuses on security and compliance within the U.S. Federal Government sector. -- **RancherD** is a new tool for installing Rancher, which is available as of Rancher v2.5.4. It is an experimental feature. RancherD is a single binary that first launches an RKE2 Kubernetes cluster, then installs the Rancher server Helm chart on the cluster. +- **RancherD** was an experimental tool for installing Rancher; a single binary that first launched an RKE2 Kubernetes cluster, then installed the Rancher server Helm chart on the cluster. It was available as part of Rancher v2.5.4 through v2.5.10 but is now deprecated and not available for recent releases. # Changes to Installation in Rancher v2.5 @@ -37,9 +37,7 @@ We recommend using Helm, a Kubernetes package manager, to install Rancher on mul ### High-availability Kubernetes Install with RancherD -_Available as of v2.5.4_ - -> This is an experimental feature. +> **Note:** RancherD was an experimental feature available as part of Rancher v2.5.4 through v2.5.10 but is now deprecated and not available for recent releases. RancherD is a single binary that first launches an RKE2 Kubernetes cluster, then installs the Rancher server Helm chart on the cluster. diff --git a/content/rancher/v2.5/en/installation/install-rancher-on-k8s/_index.md b/content/rancher/v2.5/en/installation/install-rancher-on-k8s/_index.md index 94497592e86..b0e02303a79 100644 --- a/content/rancher/v2.5/en/installation/install-rancher-on-k8s/_index.md +++ b/content/rancher/v2.5/en/installation/install-rancher-on-k8s/_index.md @@ -6,7 +6,7 @@ aliases: - /rancher/v2.5/en/installation/k8s-install/ - /rancher/v2.5/en/installation/k8s-install/helm-rancher - /rancher/v2.5/en/installation/k8s-install/kubernetes-rke - - /rancher/v2.5/en/installation/ha-server-install + - /rancher/v2.5/en/installation/ha-server-install - /rancher/v2.5/en/installation/install-rancher-on-k8s/install - /rancher/v2.x/en/installation/install-rancher-on-k8s/ --- @@ -24,7 +24,7 @@ In this section, you'll learn how to deploy Rancher on a Kubernetes cluster usin ### Kubernetes Cluster -Set up the Rancher server's local Kubernetes cluster. +Set up the Rancher server's local Kubernetes cluster. Rancher can be installed on any Kubernetes cluster. This cluster can use upstream Kubernetes, or it can use one of Rancher's Kubernetes distributions, or it can be a managed Kubernetes cluster from a provider such as Amazon EKS. @@ -113,7 +113,7 @@ There are three recommended options for the source of the certificate used for T ### 4. Install cert-manager -> You should skip this step if you are bringing your own certificate files (option `ingress.tls.source=secret`), or if you use [TLS termination on an external load balancer]({{}}/rancher/v2.5/en/installation/install-rancher-on-k8s/chart-options/#external-tls-termination). +> You should skip this step if you are bringing your own certificate files (option `ingress.tls.source=secret`), or if you use [TLS termination on an external load balancer]({{}}/rancher/v2.5/en/installation/install-rancher-on-k8s/chart-options/#external-tls-termination). This step is only required to use certificates issued by Rancher's generated CA (`ingress.tls.source=rancher`) or to request Let's Encrypt issued certificates (`ingress.tls.source=letsEncrypt`). @@ -157,6 +157,8 @@ cert-manager-webhook-787858fcdb-nlzsq 1/1 Running 0 2m The exact command to install Rancher differs depending on the certificate configuration. +However, irrespective of the certificate configuration, the name of the Rancher installation in the `cattle-system` namespace should always be `rancher`. + {{% tabs %}} {{% tab "Rancher-generated Certificates" %}} @@ -168,7 +170,7 @@ Because `rancher` is the default option for `ingress.tls.source`, we are not spe - Set `hostname` to the DNS record that resolves to your load balancer. - Set `replicas` to the number of replicas to use for the Rancher Deployment. This defaults to 3; if you have less than 3 nodes in your cluster you should reduce it accordingly. - To install a specific Rancher version, use the `--version` flag, example: `--version 2.3.6`. -- If you are installing an alpha version, Helm requires adding the `--devel` option to the command. +- If you are installing an alpha version, Helm requires adding the `--devel` option to the command. ``` helm install rancher rancher-/rancher \ @@ -190,7 +192,7 @@ deployment "rancher" successfully rolled out This option uses `cert-manager` to automatically request and renew [Let's Encrypt](https://letsencrypt.org/) certificates. This is a free service that provides you with a valid certificate as Let's Encrypt is a trusted CA. ->**Note:**: You need to have port 80 open as the HTTP-01 challenge can only be done on port 80. +>**Note:** You need to have port 80 open as the HTTP-01 challenge can only be done on port 80. In the following command, @@ -198,8 +200,9 @@ In the following command, - Set `replicas` to the number of replicas to use for the Rancher Deployment. This defaults to 3; if you have less than 3 nodes in your cluster you should reduce it accordingly. - Set `ingress.tls.source` to `letsEncrypt`. - Set `letsEncrypt.email` to the email address used for communication about your certificate (for example, expiry notices). +- Set `letsEncrypt.ingress.class` to whatever your ingress controller is, e.g., `traefik`, `nginx`, `haproxy`, etc. - To install a specific Rancher version, use the `--version` flag, example: `--version 2.3.6`. -- If you are installing an alpha version, Helm requires adding the `--devel` option to the command. +- If you are installing an alpha version, Helm requires adding the `--devel` option to the command. ``` helm install rancher rancher-/rancher \ @@ -207,7 +210,8 @@ helm install rancher rancher-/rancher \ --set hostname=rancher.my.org \ --set replicas=3 \ --set ingress.tls.source=letsEncrypt \ - --set letsEncrypt.email=me@example.org + --set letsEncrypt.email=me@example.org \ + --set letsEncrypt.ingress.class=nginx ``` Wait for Rancher to be rolled out: @@ -232,7 +236,7 @@ Although an entry in the `Subject Alternative Names` is technically required, ha - Set `replicas` to the number of replicas to use for the Rancher Deployment. This defaults to 3; if you have less than 3 nodes in your cluster you should reduce it accordingly. - Set `ingress.tls.source` to `secret`. - To install a specific Rancher version, use the `--version` flag, example: `--version 2.3.6`. -- If you are installing an alpha version, Helm requires adding the `--devel` option to the command. +- If you are installing an alpha version, Helm requires adding the `--devel` option to the command. ``` helm install rancher rancher-/rancher \ @@ -245,7 +249,7 @@ helm install rancher rancher-/rancher \ If you are using a Private CA signed certificate , add `--set privateCA=true` to the command: ``` -helm install rancher rancher-latest/rancher \ +helm install rancher rancher-/rancher \ --namespace cattle-system \ --set hostname=rancher.my.org \ --set ingress.tls.source=secret \ diff --git a/content/rancher/v2.5/en/installation/install-rancher-on-k8s/chart-options/_index.md b/content/rancher/v2.5/en/installation/install-rancher-on-k8s/chart-options/_index.md index 5f922b37e88..e1572af146b 100644 --- a/content/rancher/v2.5/en/installation/install-rancher-on-k8s/chart-options/_index.md +++ b/content/rancher/v2.5/en/installation/install-rancher-on-k8s/chart-options/_index.md @@ -124,7 +124,7 @@ To customize or use a different ingress with Rancher server you can set your own Example on setting a custom certificate issuer: ```plain ---set ingress.extraAnnotations.'certmanager\.k8s\.io/cluster-issuer'=ca-key-pair +--set ingress.extraAnnotations.'cert-manager\.io/cluster-issuer'=issuer-name ``` Example on setting a static proxy header with `ingress.configurationSnippet`. This value is parsed like a template so variables can be used. diff --git a/content/rancher/v2.5/en/installation/install-rancher-on-k8s/upgrades/_index.md b/content/rancher/v2.5/en/installation/install-rancher-on-k8s/upgrades/_index.md index 49ce4d505fc..8e1acae6ce0 100644 --- a/content/rancher/v2.5/en/installation/install-rancher-on-k8s/upgrades/_index.md +++ b/content/rancher/v2.5/en/installation/install-rancher-on-k8s/upgrades/_index.md @@ -165,7 +165,7 @@ helm upgrade rancher rancher-/rancher \ ### Option B: Reinstalling Rancher and cert-manager -If you are currently running the cert-manger whose version is older than v0.11, and want to upgrade both Rancher and cert-manager to a newer version, then you need to reinstall both Rancher and cert-manger due to the API change in cert-manger v0.11. +If you are currently running the cert-manager whose version is older than v0.11, and want to upgrade both Rancher and cert-manager to a newer version, then you need to reinstall both Rancher and cert-manager due to the API change in cert-manager v0.11. 1. Uninstall Rancher diff --git a/content/rancher/v2.5/en/installation/other-installation-methods/air-gap/install-rancher/_index.md b/content/rancher/v2.5/en/installation/other-installation-methods/air-gap/install-rancher/_index.md index d97e2266801..78daa4f58a0 100644 --- a/content/rancher/v2.5/en/installation/other-installation-methods/air-gap/install-rancher/_index.md +++ b/content/rancher/v2.5/en/installation/other-installation-methods/air-gap/install-rancher/_index.md @@ -115,7 +115,8 @@ helm template cert-manager ./cert-manager-v1.5.1.tgz --output-dir . \ --namespace cert-manager \ --set image.repository=/quay.io/jetstack/cert-manager-controller \ --set webhook.image.repository=/quay.io/jetstack/cert-manager-webhook \ - --set cainjector.image.repository=/quay.io/jetstack/cert-manager-cainjector + --set cainjector.image.repository=/quay.io/jetstack/cert-manager-cainjector \ + --set startupapicheck.image.repository=/quay.io/jetstack/cert-manager-ctl ``` ### 4. Download the cert-manager CRD diff --git a/content/rancher/v2.5/en/installation/other-installation-methods/air-gap/populate-private-registry/_index.md b/content/rancher/v2.5/en/installation/other-installation-methods/air-gap/populate-private-registry/_index.md index d42ee6f04e8..5c152feb541 100644 --- a/content/rancher/v2.5/en/installation/other-installation-methods/air-gap/populate-private-registry/_index.md +++ b/content/rancher/v2.5/en/installation/other-installation-methods/air-gap/populate-private-registry/_index.md @@ -66,7 +66,7 @@ In a Kubernetes Install, if you elect to use the Rancher default self-signed TLS helm repo add jetstack https://charts.jetstack.io helm repo update helm fetch jetstack/cert-manager --version v1.5.1 - helm template ./cert-manager-.tgz | grep -oP '(?<=image: ").*(?=")' >> ./rancher-images.txt + helm template ./cert-manager-.tgz | awk '$1 ~ /image:/ {print $2}' | sed s/\"//g >> ./rancher-images.txt ``` 2. Sort and unique the images list to remove any overlap between the sources: @@ -237,7 +237,7 @@ The workstation must have Docker 18.02+ in order to support manifests, which are helm repo add jetstack https://charts.jetstack.io helm repo update helm fetch jetstack/cert-manager --version v0.12.0 - helm template ./cert-manager-.tgz | grep -oP '(?<=image: ").*(?=")' >> ./rancher-images.txt + helm template ./cert-manager-.tgz | | awk '$1 ~ /image:/ {print $2}' | sed s/\"//g >> ./rancher-images.txt ``` 2. Sort and unique the images list to remove any overlap between the sources: diff --git a/content/rancher/v2.5/en/installation/other-installation-methods/air-gap/prepare-nodes/_index.md b/content/rancher/v2.5/en/installation/other-installation-methods/air-gap/prepare-nodes/_index.md index 4879223c572..84bff627287 100644 --- a/content/rancher/v2.5/en/installation/other-installation-methods/air-gap/prepare-nodes/_index.md +++ b/content/rancher/v2.5/en/installation/other-installation-methods/air-gap/prepare-nodes/_index.md @@ -145,7 +145,7 @@ For a how-to guide for setting up a DNS record to route domain traffic to an Ama Rancher supports air gap installs using a secure Docker private registry. You must have your own private registry or other means of distributing Docker images to your machines. -In a later step, when you set up your K3s Kubernetes cluster, you will create a [private registries configuration file]({{}}/k3s/latest/en/installation/private-registry/) with details from this registry. +In a later step, when you set up your RKE Kubernetes cluster, you will create a [private registries configuration file]({{}}/rke/latest/en/config-options/private-registries/) with details from this registry. If you need help with creating a private registry, please refer to the [official Docker documentation.](https://docs.docker.com/registry/deploying/#run-an-externally-accessible-registry) @@ -167,8 +167,6 @@ For an example of one way to set up Linux nodes, refer to this [tutorial]({{}}/k3s/latest/en/installation/private-registry/) with details from this registry. - If you need help with creating a private registry, please refer to the [official Docker documentation.](https://docs.docker.com/registry/) {{% /tab %}} diff --git a/content/rancher/v2.5/en/installation/other-installation-methods/behind-proxy/install-rancher/_index.md b/content/rancher/v2.5/en/installation/other-installation-methods/behind-proxy/install-rancher/_index.md index fac08ab40c9..dfc9854ee4d 100644 --- a/content/rancher/v2.5/en/installation/other-installation-methods/behind-proxy/install-rancher/_index.md +++ b/content/rancher/v2.5/en/installation/other-installation-methods/behind-proxy/install-rancher/_index.md @@ -26,14 +26,14 @@ kubectl create namespace cert-manager Install the CustomResourceDefinitions of cert-manager: ``` -kubectl apply --validate=false -f https://github.com/jetstack/cert-manager/releases/download/v0.15.2/cert-manager.crds.yaml +kubectl apply --validate=false -f https://github.com/jetstack/cert-manager/releases/download/v1.5.1/cert-manager.crds.yaml ``` And install it with Helm. Note that cert-manager also needs your proxy configured in case it needs to communicate with Let's Encrypt or other external certificate issuers: ``` helm upgrade --install cert-manager jetstack/cert-manager \ - --namespace cert-manager --version v0.15.2 \ + --namespace cert-manager --version v1.5.1 \ --set http_proxy=http://${proxy_host} \ --set https_proxy=http://${proxy_host} \ --set noProxy=127.0.0.0/8\\,10.0.0.0/8\\,cattle-system.svc\\,172.16.0.0/12\\,192.168.0.0/16\\,.svc\\,.cluster.local @@ -60,14 +60,16 @@ Create a namespace: kubectl create namespace cattle-system ``` -And install Rancher with Helm. Rancher also needs a proxy configuration so that it can communicate with external application catalogs or retrieve Kubernetes version update metadata: +And install Rancher with Helm. Rancher also needs a proxy configuration so that it can communicate with external application catalogs or retrieve Kubernetes version update metadata. + +Note that `rancher.cattle-system` must be added to the noProxy list (as shown below) so that Fleet can communicate directly to Rancher with Kubernetes service DNS using service discovery. ``` helm upgrade --install rancher rancher-latest/rancher \ --namespace cattle-system \ --set hostname=rancher.example.com \ --set proxy=http://${proxy_host} - --set noProxy=127.0.0.0/8\\,10.0.0.0/8\\,cattle-system.svc\\,172.16.0.0/12\\,192.168.0.0/16\\,.svc\\,.cluster.local + --set noProxy=127.0.0.0/8\\,10.0.0.0/8\\,cattle-system.svc\\,172.16.0.0/12\\,192.168.0.0/16\\,.svc\\,.cluster.local,rancher.cattle-system ``` After waiting for the deployment to finish: diff --git a/content/rancher/v2.5/en/installation/other-installation-methods/install-rancher-on-linux/_index.md b/content/rancher/v2.5/en/installation/other-installation-methods/install-rancher-on-linux/_index.md index 8b3d6e7b472..19c6afd9c62 100644 --- a/content/rancher/v2.5/en/installation/other-installation-methods/install-rancher-on-linux/_index.md +++ b/content/rancher/v2.5/en/installation/other-installation-methods/install-rancher-on-linux/_index.md @@ -6,9 +6,7 @@ aliases: - /rancher/v2.x/en/installation/install-rancher-on-linux/ --- -_Available as of Rancher v2.5.4_ - -> This is an experimental feature. +> **Note:** RancherD was an experimental feature available as part of Rancher v2.5.4 through v2.5.10 but is now deprecated and not available for recent releases. We are excited to introduce a new, simpler way to install Rancher called RancherD. diff --git a/content/rancher/v2.5/en/installation/other-installation-methods/install-rancher-on-linux/rancherd-configuration/_index.md b/content/rancher/v2.5/en/installation/other-installation-methods/install-rancher-on-linux/rancherd-configuration/_index.md index 770326f9bd7..01986082057 100644 --- a/content/rancher/v2.5/en/installation/other-installation-methods/install-rancher-on-linux/rancherd-configuration/_index.md +++ b/content/rancher/v2.5/en/installation/other-installation-methods/install-rancher-on-linux/rancherd-configuration/_index.md @@ -6,7 +6,7 @@ aliases: - /rancher/v2.x/en/installation/install-rancher-on-linux/rancherd-configuration/ --- -> RancherD is an experimental feature. +> **Note:** RancherD was an experimental feature available as part of Rancher v2.5.4 through v2.5.10 but is now deprecated and not available for recent releases. In RancherD, a server node is defined as a machine (bare-metal or virtual) running the `rancherd server` command. The server runs the Kubernetes API as well as Kubernetes workloads. @@ -22,7 +22,7 @@ In the RancherD installation instructions, we recommend running three server nod # Certificates for the Rancher Server -Rancherd does not use cert-manger to provision certs. Instead RancherD allows you to bring your own self-signed or trusted certs by storing the .pem files in `/etc/rancher/ssl/`. When doing this you should also set the `publicCA` parameter to `true` in your HelmChartConfig. For more information on the HelmChartConfig, refer to the section about [customizing the RancherD Helm chart.](#customizing-the-rancherd-helm-chart) +Rancherd does not use cert-manager to provision certs. Instead RancherD allows you to bring your own self-signed or trusted certs by storing the .pem files in `/etc/rancher/ssl/`. When doing this you should also set the `publicCA` parameter to `true` in your HelmChartConfig. For more information on the HelmChartConfig, refer to the section about [customizing the RancherD Helm chart.](#customizing-the-rancherd-helm-chart) Private key: `/etc/rancher/ssl/key.pem` diff --git a/content/rancher/v2.5/en/installation/other-installation-methods/install-rancher-on-linux/rollbacks/_index.md b/content/rancher/v2.5/en/installation/other-installation-methods/install-rancher-on-linux/rollbacks/_index.md index d9144867731..01ade576014 100644 --- a/content/rancher/v2.5/en/installation/other-installation-methods/install-rancher-on-linux/rollbacks/_index.md +++ b/content/rancher/v2.5/en/installation/other-installation-methods/install-rancher-on-linux/rollbacks/_index.md @@ -6,6 +6,6 @@ aliases: - /rancher/v2.x/en/installation/install-rancher-on-linux/rollbacks/ --- -> RancherD is an experimental feature. +> **Note:** RancherD was an experimental feature available as part of Rancher v2.5.4 through v2.5.10 but is now deprecated and not available for recent releases. To roll back Rancher to a previous version, re-run the installation script with the previous version specified in the `INSTALL_RANCHERD_VERSION` environment variable. \ No newline at end of file diff --git a/content/rancher/v2.5/en/installation/other-installation-methods/install-rancher-on-linux/upgrades/_index.md b/content/rancher/v2.5/en/installation/other-installation-methods/install-rancher-on-linux/upgrades/_index.md index 66f41139153..3f63915f6f5 100644 --- a/content/rancher/v2.5/en/installation/other-installation-methods/install-rancher-on-linux/upgrades/_index.md +++ b/content/rancher/v2.5/en/installation/other-installation-methods/install-rancher-on-linux/upgrades/_index.md @@ -6,7 +6,7 @@ aliases: - /rancher/v2.x/en/installation/install-rancher-on-linux/upgrades/ --- -> RancherD is an experimental feature. +> **Note:** RancherD was an experimental feature available as part of Rancher v2.5.4 through v2.5.10 but is now deprecated and not available for recent releases. When RancherD is upgraded, the Rancher Helm controller and the Fleet pods are upgraded. diff --git a/content/rancher/v2.5/en/installation/other-installation-methods/single-node-docker/single-node-upgrades/_index.md b/content/rancher/v2.5/en/installation/other-installation-methods/single-node-docker/single-node-upgrades/_index.md index ce5f6c3f6c2..6c55386d77d 100644 --- a/content/rancher/v2.5/en/installation/other-installation-methods/single-node-docker/single-node-upgrades/_index.md +++ b/content/rancher/v2.5/en/installation/other-installation-methods/single-node-docker/single-node-upgrades/_index.md @@ -87,7 +87,7 @@ During upgrade, you create a copy of the data from your current Rancher containe ``` - docker run --volumes-from rancher-data -v $PWD:/backup busybox tar zcvf /backup/rancher-data-backup--.tar.gz /var/lib/rancher + docker run --volumes-from rancher-data -v "$PWD:/backup" --rm busybox tar zcvf /backup/rancher-data-backup--.tar.gz /var/lib/rancher ``` **Step Result:** When you enter this command, a series of commands should run. diff --git a/content/rancher/v2.5/en/installation/requirements/_index.md b/content/rancher/v2.5/en/installation/requirements/_index.md index 77d8abc2244..41eda67564e 100644 --- a/content/rancher/v2.5/en/installation/requirements/_index.md +++ b/content/rancher/v2.5/en/installation/requirements/_index.md @@ -80,7 +80,7 @@ If you are installing Rancher on a K3s cluster with Alpine Linux, follow [these ### RancherD Specific Requirements -_The RancherD install is available as of v2.5.4. It is an experimental feature._ +> **Note:** RancherD was an experimental feature available as part of Rancher v2.5.4 through v2.5.10 but is now deprecated and not available for recent releases. At this time, only Linux OSes that leverage systemd are supported. @@ -103,6 +103,7 @@ The Ingress should be deployed as DaemonSet to ensure your load balancer can suc Docker is required for Helm chart installs, and it can be installed by following the steps in the official [Docker documentation.](https://docs.docker.com/) Rancher also provides [scripts]({{}}/rancher/v2.5/en/installation/requirements/installing-docker) to install Docker with one command. Docker is not required for RancherD installs. + # Hardware Requirements The following sections describe the CPU, memory, and disk requirements for the nodes where the Rancher server is installed. @@ -125,7 +126,7 @@ These requirements apply to RKE Kubernetes clusters, as well as to hosted Kubern | X-Large | Up to 1000 | Up to 10,000 | 16 | 64 GB | | XX-Large | Up to 2000 | Up to 20,000 | 32 | 128 GB | -[Contact Rancher](https://rancher.com/contact/) for more than 2000 clusters and/or 20,000 nodes. +Every use case and environment is different. Please [contact Rancher](https://rancher.com/contact/) to review yours. ### K3s Kubernetes @@ -139,11 +140,11 @@ These CPU and memory requirements apply to each host in a [K3s Kubernetes cluste | X-Large | Up to 1000 | Up to 10,000 | 16 | 64 GB | 2 cores, 4 GB + 1000 IOPS | | XX-Large | Up to 2000 | Up to 20,000 | 32 | 128 GB | 2 cores, 4 GB + 1000 IOPS | -[Contact Rancher](https://rancher.com/contact/) for more than 2000 clusters and/or 20,000 nodes. +Every use case and environment is different. Please [contact Rancher](https://rancher.com/contact/) to review yours. ### RancherD -_RancherD is available as of v2.5.4. It is an experimental feature._ +> **Note:** RancherD was an experimental feature available as part of Rancher v2.5.4 through v2.5.10 but is now deprecated and not available for recent releases. These CPU and memory requirements apply to each instance with RancherD installed. Minimum recommendations are outlined here. @@ -176,7 +177,7 @@ Each node in the Kubernetes cluster that Rancher is installed on should run an I The Ingress should be deployed as DaemonSet to ensure your load balancer can successfully route traffic to all nodes. -For RKE, K3s and RancherD installations, you don't have to install the Ingress manually because is is installed by default. +For RKE, K3s and RancherD installations, you don't have to install the Ingress manually because it is installed by default. For hosted Kubernetes clusters (EKS, GKE, AKS) and RKE2 Kubernetes installations, you will need to set up the ingress. diff --git a/content/rancher/v2.5/en/installation/requirements/installing-docker/_index.md b/content/rancher/v2.5/en/installation/requirements/installing-docker/_index.md index 02a005d245f..8b16d157505 100644 --- a/content/rancher/v2.5/en/installation/requirements/installing-docker/_index.md +++ b/content/rancher/v2.5/en/installation/requirements/installing-docker/_index.md @@ -3,7 +3,7 @@ title: Installing Docker weight: 1 --- -For Helm CLI installs, Docker is required to be installed on any node that runs the Rancher server. +Docker is required to be installed on nodes where the Rancher server will be installed with Helm or Docker. There are a couple of options for installing Docker. One option is to refer to the [official Docker documentation](https://docs.docker.com/install/) about how to install Docker on Linux. The steps will vary based on the Linux distribution. diff --git a/content/rancher/v2.5/en/installation/requirements/ports/_index.md b/content/rancher/v2.5/en/installation/requirements/ports/_index.md index fb88592b0ca..af7f109a891 100644 --- a/content/rancher/v2.5/en/installation/requirements/ports/_index.md +++ b/content/rancher/v2.5/en/installation/requirements/ports/_index.md @@ -119,6 +119,8 @@ The following tables break down the port requirements for inbound and outbound t ### Ports for Rancher Server Nodes on RancherD or RKE2 +> **Note:** RancherD was an experimental feature available as part of Rancher v2.5.4 through v2.5.10 but is now deprecated and not available for recent releases. + {{% accordion label="Click to expand" %}} The RancherD (or RKE2) server needs port 6443 and 9345 to be accessible by other nodes in the cluster. diff --git a/content/rancher/v2.5/en/installation/resources/advanced/firewall/_index.md b/content/rancher/v2.5/en/installation/resources/advanced/firewall/_index.md index b779951aa7b..2ff27022f24 100644 --- a/content/rancher/v2.5/en/installation/resources/advanced/firewall/_index.md +++ b/content/rancher/v2.5/en/installation/resources/advanced/firewall/_index.md @@ -5,7 +5,7 @@ aliases: - /rancher/v2.x/en/installation/resources/advanced/firewall/ --- -> We recommend disabling firewalld. For Kubernetes 1.19, firewalld must be turned off. +> We recommend disabling firewalld. For Kubernetes 1.19.x and higher, firewalld must be turned off. Some distributions of Linux [derived from RHEL,](https://en.wikipedia.org/wiki/Red_Hat_Enterprise_Linux#Rebuilds) including Oracle Linux, may have default firewall rules that block communication with Helm. diff --git a/content/rancher/v2.5/en/installation/resources/choosing-version/_index.md b/content/rancher/v2.5/en/installation/resources/choosing-version/_index.md index 9d7ec763c11..618e8e36c33 100644 --- a/content/rancher/v2.5/en/installation/resources/choosing-version/_index.md +++ b/content/rancher/v2.5/en/installation/resources/choosing-version/_index.md @@ -14,6 +14,8 @@ For Docker installations of Rancher, which is used for development and testing, The Helm chart version also applies to RancherD installs because RancherD installs the Rancher Helm chart on a Kubernetes cluster. +> **Note:** RancherD was an experimental feature available as part of Rancher v2.5.4 through v2.5.10 but is now deprecated and not available for recent releases. + {{% tabs %}} {{% tab "Helm Charts" %}} diff --git a/content/rancher/v2.5/en/installation/resources/feature-flags/_index.md b/content/rancher/v2.5/en/installation/resources/feature-flags/_index.md index d134423b4cb..b06257e74e4 100644 --- a/content/rancher/v2.5/en/installation/resources/feature-flags/_index.md +++ b/content/rancher/v2.5/en/installation/resources/feature-flags/_index.md @@ -56,7 +56,7 @@ When you install Rancher, enable the feature you want with a feature flag. The c > **Note:** Values set from the Rancher API will override the value passed in through the command line. -When installing Rancher with a Helm chart, use the `--features` option. In the below example, two features are enabled by passing the feature flag names names in a comma separated list: +When installing Rancher with a Helm chart, use the `--set` option. In the below example, two features are enabled by passing the feature flag names in a comma separated list: ``` helm install rancher-latest/rancher \ @@ -131,7 +131,7 @@ When installing Rancher with Docker, use the `--features` option. In the below e docker run -d -p 80:80 -p 443:443 \ --restart=unless-stopped \ rancher/rancher:rancher-latest \ - --features==true,=true + --features==true,=true ``` diff --git a/content/rancher/v2.5/en/installation/resources/installing-docker/_index.md b/content/rancher/v2.5/en/installation/resources/installing-docker/_index.md deleted file mode 100644 index ceb75b12645..00000000000 --- a/content/rancher/v2.5/en/installation/resources/installing-docker/_index.md +++ /dev/null @@ -1,20 +0,0 @@ ---- -title: Installing Docker -weight: 1 -aliases: - - /rancher/v2.5/en/installation/requirements/installing-docker ---- - -Docker is required to be installed on nodes where the Rancher server will be installed with Helm or Docker. Docker is not required for RancherD installs. - -There are a couple of options for installing Docker. One option is to refer to the [official Docker documentation](https://docs.docker.com/install/) about how to install Docker on Linux. The steps will vary based on the Linux distribution. - -Another option is to use one of Rancher's Docker installation scripts, which are available for most recent versions of Docker. - -For example, this command could be used to install Docker 19.03 on Ubuntu: - -``` -curl https://releases.rancher.com/install-docker/19.03.sh | sh -``` - -Rancher has installation scripts for every version of upstream Docker that Kubernetes supports. To find out whether a script is available for installing a certain Docker version, refer to this [GitHub repository,](https://github.com/rancher/install-docker) which contains all of Rancher's Docker installation scripts. \ No newline at end of file diff --git a/content/rancher/v2.5/en/installation/resources/k8s-tutorials/ha-rke2/_index.md b/content/rancher/v2.5/en/installation/resources/k8s-tutorials/ha-rke2/_index.md index 64d7472dbdd..788a80c1e4a 100644 --- a/content/rancher/v2.5/en/installation/resources/k8s-tutorials/ha-rke2/_index.md +++ b/content/rancher/v2.5/en/installation/resources/k8s-tutorials/ha-rke2/_index.md @@ -8,7 +8,7 @@ aliases: --- _Tested on v2.5.6_ -This section describes how to install a Kubernetes cluster according to the [best practices for the Rancher server environment.]({{}}/rancher/v2.x/en/overview/architecture-recommendations/#environment-for-kubernetes-installations) +This section describes how to install a Kubernetes cluster according to the [best practices for the Rancher server environment.]({{}}/rancher/v2.5/en/overview/architecture-recommendations/#environment-for-kubernetes-installations) # Prerequisites diff --git a/content/rancher/v2.5/en/installation/resources/k8s-tutorials/infrastructure-tutorials/infra-for-rke2-ha/_index.md b/content/rancher/v2.5/en/installation/resources/k8s-tutorials/infrastructure-tutorials/infra-for-rke2-ha/_index.md index ac26cf28e2a..0f4d80d55c1 100644 --- a/content/rancher/v2.5/en/installation/resources/k8s-tutorials/infrastructure-tutorials/infra-for-rke2-ha/_index.md +++ b/content/rancher/v2.5/en/installation/resources/k8s-tutorials/infrastructure-tutorials/infra-for-rke2-ha/_index.md @@ -19,9 +19,9 @@ To install the Rancher management server on a high-availability RKE2 cluster, we ### 1. Set up Linux Nodes -Make sure that your nodes fulfill the general installation requirements for [OS, container runtime, hardware, and networking.]({{}}/rancher/v2.x/en/installation/requirements/) +Make sure that your nodes fulfill the general installation requirements for [OS, container runtime, hardware, and networking.]({{}}/rancher/v2.5/en/installation/requirements/) -For an example of one way to set up Linux nodes, refer to this [tutorial]({{}}/rancher/v2.x/en/installation/options/ec2-node) for setting up nodes as instances in Amazon EC2. +For an example of one way to set up Linux nodes, refer to this [tutorial]({{}}/rancher/v2.5/en/installation/options/ec2-node) for setting up nodes as instances in Amazon EC2. ### 2. Set up the Load Balancer @@ -34,11 +34,11 @@ When Rancher is installed (also in a later step), the Rancher system creates an For your implementation, consider if you want or need to use a Layer-4 or Layer-7 load balancer: - **A layer-4 load balancer** is the simpler of the two choices, in which you are forwarding TCP traffic to your nodes. We recommend configuring your load balancer as a Layer 4 balancer, forwarding traffic to ports TCP/80 and TCP/443 to the Rancher management cluster nodes. The Ingress controller on the cluster will redirect HTTP traffic to HTTPS and terminate SSL/TLS on port TCP/443. The Ingress controller will forward traffic to port TCP/80 to the Ingress pod in the Rancher deployment. -- **A layer-7 load balancer** is a bit more complicated but can offer features that you may want. For instance, a layer-7 load balancer is capable of handling TLS termination at the load balancer, as opposed to Rancher doing TLS termination itself. This can be beneficial if you want to centralize your TLS termination in your infrastructure. Layer-7 load balancing also offers the capability for your load balancer to make decisions based on HTTP attributes such as cookies, etc. that a layer-4 load balancer is not able to concern itself with. If you decide to terminate the SSL/TLS traffic on a layer-7 load balancer, you will need to use the `--set tls=external` option when installing Rancher in a later step. For more information, refer to the [Rancher Helm chart options.]({{}}/rancher/v2.x/en/installation/options/chart-options/#external-tls-termination) +- **A layer-7 load balancer** is a bit more complicated but can offer features that you may want. For instance, a layer-7 load balancer is capable of handling TLS termination at the load balancer, as opposed to Rancher doing TLS termination itself. This can be beneficial if you want to centralize your TLS termination in your infrastructure. Layer-7 load balancing also offers the capability for your load balancer to make decisions based on HTTP attributes such as cookies, etc. that a layer-4 load balancer is not able to concern itself with. If you decide to terminate the SSL/TLS traffic on a layer-7 load balancer, you will need to use the `--set tls=external` option when installing Rancher in a later step. For more information, refer to the [Rancher Helm chart options.]({{}}/rancher/v2.5/en/installation/options/chart-options/#external-tls-termination) -For an example showing how to set up an NGINX load balancer, refer to [this page.]({{}}/rancher/v2.x/en/installation/options/nginx/) +For an example showing how to set up an NGINX load balancer, refer to [this page.]({{}}/rancher/v2.5/en/installation/options/nginx/) -For a how-to guide for setting up an Amazon ELB Network Load Balancer, refer to [this page.]({{}}/rancher/v2.x/en/installation/options/nlb/) +For a how-to guide for setting up an Amazon ELB Network Load Balancer, refer to [this page.]({{}}/rancher/v2.5/en/installation/options/nlb/) > **Important:** > Do not use this load balancer (i.e, the `local` cluster Ingress) to load balance applications other than Rancher following installation. Sharing this Ingress with other applications may result in websocket errors to Rancher following Ingress configuration reloads for other apps. We recommend dedicating the `local` cluster to Rancher and no other applications. diff --git a/content/rancher/v2.5/en/installation/resources/tls-secrets/_index.md b/content/rancher/v2.5/en/installation/resources/tls-secrets/_index.md index 850856ff865..9878109b8b6 100644 --- a/content/rancher/v2.5/en/installation/resources/tls-secrets/_index.md +++ b/content/rancher/v2.5/en/installation/resources/tls-secrets/_index.md @@ -38,4 +38,4 @@ kubectl -n cattle-system create secret generic tls-ca \ # Updating a Private CA Certificate -Follow the steps on [this page]({{}}/rancher/v2.x/en/installation/resources/update-ca-cert) to update the SSL certificate of the ingress in a Rancher [high availability Kubernetes installation]({{}}/rancher/v2.5/en/installation/install-rancher-on-k8s/) or to switch from the default self-signed certificate to a custom certificate. \ No newline at end of file +Follow the steps on [this page]({{}}/rancher/v2.5/en/installation/resources/update-ca-cert) to update the SSL certificate of the ingress in a Rancher [high availability Kubernetes installation]({{}}/rancher/v2.5/en/installation/install-rancher-on-k8s/) or to switch from the default self-signed certificate to a custom certificate. \ No newline at end of file diff --git a/content/rancher/v2.5/en/installation/resources/update-ca-cert/_index.md b/content/rancher/v2.5/en/installation/resources/update-ca-cert/_index.md index c256f9f5905..751feae2d8c 100644 --- a/content/rancher/v2.5/en/installation/resources/update-ca-cert/_index.md +++ b/content/rancher/v2.5/en/installation/resources/update-ca-cert/_index.md @@ -13,6 +13,7 @@ A summary of the steps is as follows: 2. Create or update the `tls-ca` Kubernetes secret resource with the root CA certificate (only required when using a private CA). 3. Update the Rancher installation using the Helm CLI. 4. Reconfigure the Rancher agents to trust the new CA certificate. +5. Select Force Update of Fleet clusters to connect fleet-agent to Rancher. The details of these instructions are below. @@ -145,3 +146,12 @@ First, generate the agent definitions as described here: https://gist.github.com Then, connect to a controlplane node of the downstream cluster via SSH, create a Kubeconfig and apply the definitions: https://gist.github.com/superseb/b14ed3b5535f621ad3d2aa6a4cd6443b + + +# 5. Select Force Update of Fleet clusters to connect fleet-agent to Rancher + +Select 'Force Update' for the clusters within the [Continuous Delivery]({{}}/rancher/v2.5/en/deploy-across-clusters/fleet/#accessing-fleet-in-the-rancher-ui) view under Cluster Explorer in the Rancher UI to allow the fleet-agent in downstream clusters to successfully connect to Rancher. + +### Why is this step required? + +Fleet agents in Rancher managed clusters store kubeconfig that is used to connect to the Rancher proxied kube-api in the fleet-agent secret of the fleet-system namespace. The kubeconfig contains a certificate-authority-data block containing the Rancher CA. When changing the Rancher CA, this block needs to be updated for a successful connection of the fleet-agent to Rancher. diff --git a/content/rancher/v2.5/en/installation/resources/upgrading-cert-manager/_index.md b/content/rancher/v2.5/en/installation/resources/upgrading-cert-manager/_index.md index a3be656b66a..9b557f0ccc7 100644 --- a/content/rancher/v2.5/en/installation/resources/upgrading-cert-manager/_index.md +++ b/content/rancher/v2.5/en/installation/resources/upgrading-cert-manager/_index.md @@ -20,7 +20,7 @@ To address these changes, this guide will do two things: 1. Explain the cert-manager API changes and link to cert-manager's official documentation for migrating your data > **Important:** -> If you are currently running the cert-manger whose version is older than v0.11, and want to upgrade both Rancher and cert-manager to a newer version, you need to reinstall both of them: +> If you are currently running the cert-manager whose version is older than v0.11, and want to upgrade both Rancher and cert-manager to a newer version, you need to reinstall both of them: > 1. Take a one-time snapshot of your Kubernetes cluster running Rancher server > 2. Uninstall Rancher, cert-manager, and the CustomResourceDefinition for cert-manager diff --git a/content/rancher/v2.5/en/installation/resources/upgrading-cert-manager/helm-2-instructions/_index.md b/content/rancher/v2.5/en/installation/resources/upgrading-cert-manager/helm-2-instructions/_index.md index 761a2487d26..070dff28f02 100644 --- a/content/rancher/v2.5/en/installation/resources/upgrading-cert-manager/helm-2-instructions/_index.md +++ b/content/rancher/v2.5/en/installation/resources/upgrading-cert-manager/helm-2-instructions/_index.md @@ -18,7 +18,7 @@ To address these changes, this guide will do two things: 1. Explain the cert-manager API changes and link to cert-manager's offficial documentation for migrating your data > **Important:** -> If you are currently running the cert-manger whose version is older than v0.11, and want to upgrade both Rancher and cert-manager to a newer version, you need to reinstall both of them: +> If you are currently running the cert-manager whose version is older than v0.11, and want to upgrade both Rancher and cert-manager to a newer version, you need to reinstall both of them: > 1. Take a one-time snapshot of your Kubernetes cluster running Rancher server > 2. Uninstall Rancher, cert-manager, and the CustomResourceDefinition for cert-manager diff --git a/content/rancher/v2.5/en/k8s-in-rancher/workloads/deploy-workloads/_index.md b/content/rancher/v2.5/en/k8s-in-rancher/workloads/deploy-workloads/_index.md index c27ebdcc9e6..4d90d3f5db1 100644 --- a/content/rancher/v2.5/en/k8s-in-rancher/workloads/deploy-workloads/_index.md +++ b/content/rancher/v2.5/en/k8s-in-rancher/workloads/deploy-workloads/_index.md @@ -16,7 +16,7 @@ Deploy a workload to run an application in one or more containers. 1. Enter a **Name** for the workload. -1. Select a [workload type]({{}}/rancher/v2.5/en/k8s-in-rancher/workloads/). The workload defaults to a scalable deployment, by can change the workload type by clicking **More options.** +1. Select a [workload type]({{}}/rancher/v2.5/en/k8s-in-rancher/workloads/). The workload defaults to a scalable deployment, but you can change the workload type by clicking **More options.** 1. From the **Docker Image** field, enter the name of the Docker image that you want to deploy to the project, optionally prefacing it with the registry host (e.g. `quay.io`, `registry.gitlab.com`, etc.). During deployment, Rancher pulls this image from the specified public or private registry. If no registry host is provided, Rancher will pull the image from [Docker Hub](https://hub.docker.com/explore/). Enter the name exactly as it appears in the registry server, including any required path, and optionally including the desired tag (e.g. `registry.gitlab.com/user/path/image:tag`). If no tag is provided, the `latest` tag will be automatically used. diff --git a/content/rancher/v2.5/en/monitoring-alerting/configuration/_index.md b/content/rancher/v2.5/en/monitoring-alerting/configuration/_index.md index e56978fbb98..2d7867cbc4c 100644 --- a/content/rancher/v2.5/en/monitoring-alerting/configuration/_index.md +++ b/content/rancher/v2.5/en/monitoring-alerting/configuration/_index.md @@ -15,6 +15,8 @@ For information on configuring custom scrape targets and rules for Prometheus, p The resource requests and limits for the monitoring application can be configured when installing `rancher-monitoring`. For more information about the default limits, see [this page.](./helm-chart-options/#configuring-resource-limits-and-requests) +>**Note:** On an idle cluster, Monitoring V2 has significantly higher CPU usage (up to 70%) as compared to Monitoring V1. To improve performance and achieve similar results as in Monitoring V1, turn off the Prometheus adapter. + # Prometheus Configuration It is usually not necessary to directly edit the Prometheus custom resource. diff --git a/content/rancher/v2.5/en/monitoring-alerting/configuration/servicemonitor-podmonitor/_index.md b/content/rancher/v2.5/en/monitoring-alerting/configuration/servicemonitor-podmonitor/_index.md index 79f3da27930..39ddfd2b5a0 100644 --- a/content/rancher/v2.5/en/monitoring-alerting/configuration/servicemonitor-podmonitor/_index.md +++ b/content/rancher/v2.5/en/monitoring-alerting/configuration/servicemonitor-podmonitor/_index.md @@ -26,6 +26,6 @@ For more information about how ServiceMonitors work, refer to the [Prometheus Op This pseudo-CRD maps to a section of the Prometheus custom resource configuration. It declaratively specifies how group of pods should be monitored. -When a PodMonitor is created, the Prometheus Operator updates the Prometheus scrape configuration to include the PodMonitor configuration. Then Prometheus begins scraping metrics from the endpoint defined in the ServiceMonitor. +When a PodMonitor is created, the Prometheus Operator updates the Prometheus scrape configuration to include the PodMonitor configuration. Then Prometheus begins scraping metrics from the endpoint defined in the PodMonitor. Any Pods in your cluster that match the labels located within the PodMonitor `selector` field will be monitored based on the `podMetricsEndpoints` specified on the PodMonitor. For more information on what fields can be specified, please look at the [spec](https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/api.md#podmonitorspec) provided by Prometheus Operator. diff --git a/content/rancher/v2.5/en/monitoring-alerting/how-monitoring-works/_index.md b/content/rancher/v2.5/en/monitoring-alerting/how-monitoring-works/_index.md index 6445df9e75a..30b19cdbc66 100644 --- a/content/rancher/v2.5/en/monitoring-alerting/how-monitoring-works/_index.md +++ b/content/rancher/v2.5/en/monitoring-alerting/how-monitoring-works/_index.md @@ -11,52 +11,78 @@ weight: 1 # 1. Architecture Overview -This diagram shows how data flows through the Monitoring V2 application: +_**The following sections describe how data flows through the Monitoring V2 application:**_ -{{% row %}} -{{% column %}} +### Prometheus Operator -![How data flows through the monitoring application]({{}}/img/rancher/monitoring-v2-architecture-overview.svg) +Prometheus Operator observes ServiceMonitors, PodMonitors, and PrometheusRules being created. When the Prometheus configuration resources are created, Prometheus Operator calls the Prometheus API to sync the new configuration. As the diagram at the end of this section shows, the Prometheus Operator acts as the intermediary between Prometheus and Kubernetes, calling the Prometheus API to synchronize Prometheus with the monitoring-related resources in Kubernetes. -{{% /column %}} -{{% column %}} +### ServiceMonitors and PodMonitors +ServiceMonitors and PodMonitors declaratively specify targets, such as Services and Pods, that need to be monitored. -1. Rules define what Prometheus metrics or time series database queries should result in alerts being fired. -2. ServiceMonitors and PodMonitors declaratively specify how services and pods should be monitored. They use labels to scrape metrics from pods. -3. Prometheus Operator observes ServiceMonitors, PodMonitors and PrometheusRules being created. -4. When the Prometheus configuration resources are created, Prometheus Operator calls the Prometheus API to sync the new configuration. -5. Recording Rules are not directly used for alerting. They create new time series of precomputed queries. These new time series data can then be queried to generate alerts. -6. Prometheus scrapes all targets in the scrape configuration on a recurring schedule based on the scrape interval, storing the results in its time series database.Depending on the Kubernetes master component and Kubernetes distribution, the metrics from a certain Kubernetes component could be directly exposed to Prometheus, proxied through PushProx, or not available. For details, see Scraping and Exposing Metrics. -7. Prometheus evaluates the alerting rules against the time series database. It fires alerts to Alertmanager whenever an alerting rule evaluates to a positive number. -8. Alertmanager uses routes to group, label and filter the fired alerts to translate them into useful notifications. -9. Alertmanager uses the Receiver configuration to send notifications to Slack, PagerDuty, SMS, or other types of receivers. +- Targets are scraped on a recurring schedule based on the configured Prometheus scrape interval, and the metrics that are scraped are stored into the Prometheus Time Series Database (TSDB). -{{% /column %}} -{{% /row %}} +- In order to perform the scrape, ServiceMonitors and PodMonitors are defined with label selectors that determine which Services or Pods should be scraped and endpoints that determine how the scrape should happen on the given target, e.g., scrape/metrics in TCP 10252, proxying through IP addr x.x.x.x. +- Out of the box, Monitoring V2 comes with certain pre-configured exporters that are deployed based on the type of Kubernetes cluster that it is deployed on. For more information, see [Scraping and Exposing Metrics](#5-scraping-and-exposing-metrics). +### How PushProx Works + +- Certain internal Kubernetes components are scraped via a proxy deployed as part of Monitoring V2 called **PushProx**. The Kubernetes components that expose metrics to Prometheus through PushProx are the following: +`kube-controller-manager`, `kube-scheduler`, `etcd`, and `kube-proxy`. + +- For each PushProx exporter, we deploy one PushProx client onto all target nodes. For example, a PushProx client is deployed onto all controlplane nodes for kube-controller-manager, all etcd nodes for kube-etcd, and all nodes for kubelet. + +- We deploy exactly one PushProx proxy per exporter. The process for exporting metrics is as follows: + +1. The PushProx Client establishes an outbound connection with the PushProx Proxy. +1. The client then polls the proxy for scrape requests that have come into the proxy. +1. When the proxy receives a scrape request from Prometheus, the client sees it as a result of the poll. +1. The client scrapes the internal component. +1. The internal component responds by pushing metrics back to the proxy. + + +

Process for Exporting Metrics with PushProx:
+ +![Process for Exporting Metrics with PushProx]({{}}/img/rancher/pushprox-process.svg) + +### PrometheusRules + +PrometheusRules allow users to define rules for what metrics or time series database queries should result in alerts being fired. Rules are evaluated on an interval. + +- **Recording rules** create a new time series based on existing series that have been collected. They are frequently used to precompute complex queries. +- **Alerting rules** run a particular query and fire an alert from Prometheus if the query evaluates to a non-zero value. + +### Alert Routing + +Once Prometheus determines that an alert needs to be fired, alerts are forwarded to **Alertmanager**. + +- Alerts contain labels that come from the PromQL query itself and additional labels and annotations that can be provided as part of specifying the initial PrometheusRule. + +- Before receiving any alerts, Alertmanager will use the **routes** and **receivers** specified in its configuration to form a routing tree on which all incoming alerts are evaluated. Each node of the routing tree can specify additional grouping, labeling, and filtering that needs to happen based on the labels attached to the Prometheus alert. A node on the routing tree (usually a leaf node) can also specify that an alert that reaches it needs to be sent out to a configured Receiver, e.g., Slack, PagerDuty, SMS, etc. Note that Alertmanager will send an alert first to **alertingDriver**, then alertingDriver will send or forward alert to the proper destination. + +- Routes and receivers are also stored in the Kubernetes API via the Alertmanager Secret. When the Secret is updated, Alertmanager is also updated automatically. Note that routing occurs via labels only (not via annotations, etc.). + +
How data flows through the monitoring application:
# 2. How Prometheus Works -### 2.1. Storing Time Series Data +### Storing Time Series Data After collecting metrics from exporters, Prometheus stores the time series in a local on-disk time series database. Prometheus optionally integrates with remote systems, but `rancher-monitoring` uses local storage for the time series database. -The database can then be queried using PromQL, the query language for Prometheus. Grafana dashboards use PromQL queries to generate data visualizations. +Once stored, users can query this TSDB using PromQL, the query language for Prometheus. -### 2.2. Querying the Time Series Database +PromQL queries can be visualized in one of two ways: -The PromQL query language is the primary tool to query Prometheus for time series data. +1. By supplying the query in Prometheus's Graph UI, which will show a simple graphical view of the data. +1. By creating a Grafana Dashboard that contains the PromQL query and additional formatting directives that label axes, add units, change colors, use alternative visualizations, etc. -In Grafana, you can right-click a CPU utilization and click Inspect. This opens a panel that shows the [raw query results.](https://grafana.com/docs/grafana/latest/panels/inspect-panel/#inspect-raw-query-results)The raw results demonstrate how each dashboard is powered by PromQL queries. +### Defining Rules for Prometheus -### 2.3. Defining Rules for when Alerts Should be Fired - -Rules define the conditions for Prometheus to fire alerts. When PrometheusRule custom resources are created or updated, the Prometheus Operator observes the change and calls the Prometheus API to synchronize the rule configuration with the Alerting Rules and Recording Rules in Prometheus. - -When you define a Rule (which is declared within a RuleGroup in a PrometheusRule resource), the [spec of the Rule itself](https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/api.md#rule) contains labels that are used by Alertmanager to figure out which Route should receive this Alert. For example, an Alert with the label `team: front-end` will be sent to all Routes that match on that label. +Rules define queries that Prometheus needs to execute on a regular `evaluationInterval` to perform certain actions, such as firing an alert (alerting rules) or precomputing a query based on others existing in its TSDB (recording rules). These rules are encoded in PrometheusRules custom resources. When PrometheusRule custom resources are created or updated, the Prometheus Operator observes the change and calls the Prometheus API to synchronize the set of rules that Prometheus is currently evaluating on a regular interval. A PrometheusRule allows you to define one or more RuleGroups. Each RuleGroup consists of a set of Rule objects that can each represent either an alerting or a recording rule with the following fields: @@ -65,7 +91,9 @@ A PrometheusRule allows you to define one or more RuleGroups. Each RuleGroup con - Labels that should be attached to the alert or record that identify it (e.g. cluster name or severity) - Annotations that encode any additional important pieces of information that need to be displayed on the notification for an alert (e.g. summary, description, message, runbook URL, etc.). This field is not required for recording rules. -### 2.4. Firing Alerts +On evaluating a [rule](https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/api.md#rule), Prometheus will execute the provided PromQL query, add additional provided labels (or annotations - only for alerting rules), and execute the appropriate action for the rule. For example, an Alerting Rule that adds `team: front-end` as a label to the provided PromQL query will append that label to the fired alert, which will allow Alertmanager to forward the alert to the correct Receiver. + +### Alerting and Recording Rules Prometheus doesn't maintain the state of whether alerts are active. It fires alerts repetitively at every evaluation interval, relying on Alertmanager to group and filter the alerts into meaningful notifications. @@ -80,6 +108,7 @@ Alerting rules are more commonly used. Whenever an alerting rule evaluates to a The Rule file adds labels and annotations to alerts before firing them, depending on the use case: - Labels indicate information that identifies the alert and could affect the routing of the alert. For example, if when sending an alert about a certain container, the container ID could be used as a label. + - Annotations denote information that doesn't affect where an alert is routed, for example, a runbook or an error message. # 3. How Alertmanager Works @@ -87,17 +116,24 @@ The Rule file adds labels and annotations to alerts before firing them, dependin The Alertmanager handles alerts sent by client applications such as the Prometheus server. It takes care of the following tasks: - Deduplicating, grouping, and routing alerts to the correct receiver integration such as email, PagerDuty, or OpsGenie + - Silencing and inhibition of alerts + - Tracking alerts that fire over time + - Sending out the status of whether an alert is currently firing, or if it is resolved + +### Alerts Forwarded by alertingDrivers + +When alertingDrivers are installed, this creates a `Service` that can be used as the receiver's URL for Teams or SMS, based on the alertingDriver's configuration. The URL in the Receiver points to the alertingDrivers; so the Alertmanager sends alert first to alertingDriver, then alertingDriver forwards or sends alert to the proper destination. -### 3.1. Routing Alerts to Receivers +### Routing Alerts to Receivers Alertmanager coordinates where alerts are sent. It allows you to group alerts based on labels and fire them based on whether certain labels are matched. One top-level route accepts all alerts. From there, Alertmanager continues routing alerts to receivers based on whether they match the conditions of the next route. -While the Rancher UI forms only allow editing a routing tree that is two levels deep, you can configure more deeply nested routing structures by editing the Alertmanager custom resource YAML. +While the Rancher UI forms only allow editing a routing tree that is two levels deep, you can configure more deeply nested routing structures by editing the Alertmanager Secret. -### 3.2. Configuring Multiple Receivers +### Configuring Multiple Receivers By editing the forms in the Rancher UI, you can set up a Receiver resource with all the information Alertmanager needs to send alerts to your notification system. @@ -109,120 +145,89 @@ Prometheus Operator introduces a set of [Custom Resource Definitions](https://gi Prometheus Operator will automatically update your Prometheus configuration based on the live state of the resources and configuration options that are edited in the Rancher UI. -### 4.1. Resources Deployed by Default +### Resources Deployed by Default By default, a set of resources curated by the [kube-prometheus](https://github.com/prometheus-operator/kube-prometheus) project are deployed onto your cluster as part of installing the Rancher Monitoring Application to set up a basic Monitoring/Alerting stack. The resources that get deployed onto your cluster to support this solution can be found in the [`rancher-monitoring`](https://github.com/rancher/charts/tree/main/charts/rancher-monitoring) Helm chart, which closely tracks the upstream [kube-prometheus-stack](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack) Helm chart maintained by the Prometheus community with certain changes tracked in the [CHANGELOG.md](https://github.com/rancher/charts/blob/main/charts/rancher-monitoring/CHANGELOG.md). -There are also certain special types of ConfigMaps and Secrets such as those corresponding to Grafana Dashboards, Grafana Datasources, and Alertmanager Configs that will automatically update your Prometheus configuration via sidecar proxies that observe the live state of those resources within your cluster. +### Default Exporters -### 4.2. PushProx +Monitoring V2 deploys three default exporters that provide additional metrics for Prometheus to store: -PushProx enhances the security of the monitoring application, allowing it to be installed on hardened Kubernetes clusters. +1. `node-exporter`: exposes hardware and OS metrics for Linux hosts. For more information on `node-exporter`, refer to the [upstream documentation](https://prometheus.io/docs/guides/node-exporter/). -To expose Kubernetes metrics, PushProxes use a client proxy model to expose specific ports within default Kubernetes components. Node exporters expose metrics to PushProx through an outbound connection. +1. `windows-exporter`: exposes hardware and OS metrics for Windows hosts (only deployed on Windows clusters). For more information on `windows-exporter`, refer to the [upstream documentation](https://github.com/prometheus-community/windows_exporter). -The proxy allows `rancher-monitoring` to scrape metrics from processes on the hostNetwork, such as the `kube-api-server`, without opening up node ports to inbound connections. +1. `kube-state-metrics`: expose additional metrics that track the state of resources contained in the Kubernetes API (e.g., pods, workloads, etc.). For more information on `kube-state-metrics`, refer to the [upstream documentation](https://github.com/kubernetes/kube-state-metrics/tree/master/docs). -PushProx is a DaemonSet that listens for clients that seek to register. Once registered, it proxies scrape requests through the established connection. Then the client executes the request to etcd. +ServiceMonitors and PodMonitors will scrape these exporters, as defined [here](#defining-what-metrics-are-scraped). Prometheus stores these metrics, and you can query the results via either Prometheus's UI or Grafana. -All of the default ServiceMonitors, such as `rancher-monitoring-kube-controller-manager`, are configured to hit the metrics endpoint of the client using this proxy. +See the [architecture](#1-architecture-overview) section for more information on recording rules, alerting rules, and Alertmanager. -For more details about how PushProx works, refer to [Scraping Metrics with PushProx.](#5-5-scraping-metrics-with-pushprox) - - -### 4.3. Default Exporters - -`rancher-monitoring` deploys two exporters to expose metrics to prometheus: `node-exporter` and `windows-exporter`. Both are deployed as DaemonSets. - -`node-exporter` exports container, pod and node metrics for CPU and memory from each Linux node. `windows-exporter` does the same, but for Windows nodes. - -For more information on `node-exporter`, refer to the [upstream documentation.](https://prometheus.io/docs/guides/node-exporter/) - -[kube-state-metrics](https://github.com/kubernetes/kube-state-metrics) is also useful because it exports metrics for Kubernetes components. - -### 4.4. Components Exposed in the Rancher UI +### Components Exposed in the Rancher UI When the monitoring application is installed, you will be able to edit the following components in the Rancher UI: | Component | Type of Component | Purpose and Common Use Cases for Editing | |--------------|------------------------|---------------------------| -| ServiceMonitor | Custom resource | Set up targets to scrape custom metrics from. Automatically updates the scrape configuration in the Prometheus custom resource. | -| PodMonitor | Custom resource | Set up targets to scrape custom metrics from. Automatically updates the scrape configuration in the Prometheus custom resource. | -| Receiver | Configuration block (part of Alertmanager) | Set up a notification system to receive alerts. Automatically updates the Alertmanager custom resource. | -| Route | Configuration block (part of Alertmanager) | Add identifying information to make alerts more meaningful and direct them to individual teams. Automatically updates the Alertmanager custom resource. | -| PrometheusRule | Custom resource | For more advanced use cases, you may want to define what Prometheus metrics or time series database queries should result in alerts being fired. Automatically updates the Prometheus custom resource. | -| Alertmanager | Custom resource | Edit this custom resource only if you need more advanced configuration options beyond what the Rancher UI exposes in the Routes and Receivers sections. For example, you might want to edit this resource to add a routing tree with more than two levels. | -| Prometheus | Custom resource | Edit this custom resource only if you need more advanced configuration beyond what can be configured using ServiceMonitors, PodMonitors, or [Rancher monitoring Helm chart options.](../configuration/helm-chart-options) | +| ServiceMonitor | Custom resource | Sets up Kubernetes Services to scrape custom metrics from. Automatically updates the scrape configuration in the Prometheus custom resource. | +| PodMonitor | Custom resource | Sets up Kubernetes Pods to scrape custom metrics from. Automatically updates the scrape configuration in the Prometheus custom resource. | +| Receiver | Configuration block (part of Alertmanager) | Modifies information on where to send an alert (e.g., Slack, PagerDuty, etc.) and any necessary information to send the alert (e.g., TLS certs, proxy URLs, etc.). Automatically updates the Alertmanager custom resource. | +| Route | Configuration block (part of Alertmanager) | Modifies the routing tree that is used to filter, label, and group alerts based on labels and send them to the appropriate Receiver. Automatically updates the Alertmanager custom resource. | +| PrometheusRule | Custom resource | Defines additional queries that need to trigger alerts or define materialized views of existing series that are within Prometheus's TSDB. Automatically updates the Prometheus custom resource. | + +### PushProx + +PushProx allows Prometheus to scrape metrics across a network boundary, which prevents users from having to expose metrics ports for internal Kubernetes components on each node in a Kubernetes cluster. + +Since the metrics for Kubernetes components are generally exposed on the host network of nodes in the cluster, PushProx deploys a DaemonSet of clients that sit on the hostNetwork of each node and make an outbound connection to a single proxy that is sitting on the Kubernetes API. Prometheus can then be configured to proxy scrape requests through the proxy to each client, which allows it to scrape metrics from the internal Kubernetes components without requiring any inbound node ports to be open. + +Refer to [Scraping Metrics with PushProx](#scraping-metrics-with-pushprox) for more. # 5. Scraping and Exposing Metrics -### 5.1. Defining what Metrics are Scraped +### Defining what Metrics are Scraped -ServiceMonitors define targets that are intended for Prometheus to scrape. The [Prometheus custom resource tells](https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/design.md#prometheus) Prometheus which ServiceMonitors it should use to find out where to scrape metrics from. +ServiceMonitors and PodMonitors define targets that are intended for Prometheus to scrape. The [Prometheus custom resource](https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/design.md#prometheus) tells Prometheus which ServiceMonitors or PodMonitors it should use to find out where to scrape metrics from. -The Prometheus Operator observes the ServiceMonitors. When it observes that ServiceMonitors are created or updated, it calls the Prometheus API to update the scrape configuration in the Prometheus custom resource and keep it in sync with the scrape configuration in the ServiceMonitors. This scrape configuration tells Prometheus which endpoints to scrape metrics from and how it will label the metrics from those endpoints. +The Prometheus Operator observes the ServiceMonitors and PodMonitors. When it observes that they are created or updated, it calls the Prometheus API to update the scrape configuration in the Prometheus custom resource and keep it in sync with the scrape configuration in the ServiceMonitors or PodMonitors. This scrape configuration tells Prometheus which endpoints to scrape metrics from and how it will label the metrics from those endpoints. Prometheus scrapes all of the metrics defined in its scrape configuration at every `scrape_interval`, which is one minute by default. The scrape configuration can be viewed as part of the Prometheus custom resource that is exposed in the Rancher UI. -### 5.2. How the Prometheus Operator Sets up Metrics Scraping +### How the Prometheus Operator Sets up Metrics Scraping The Prometheus Deployment or StatefulSet scrapes metrics, and the configuration of Prometheus is controlled by the Prometheus custom resources. The Prometheus Operator watches for Prometheus and Alertmanager resources, and when they are created, the Prometheus Operator creates a Deployment or StatefulSet for Prometheus or Alertmanager with the user-defined configuration. -
How the Prometheus Operator Sets up Metrics Scraping
+When the Prometheus Operator observes ServiceMonitors, PodMonitors, and PrometheusRules being created, it knows that the scrape configuration needs to be updated in Prometheus. It updates Prometheus by first updating the configuration and rules files in the volumes of Prometheus's Deployment or StatefulSet. Then it calls the Prometheus API to sync the new configuration, resulting in the Prometheus Deployment or StatefulSet to be modified in place. -![How the Prometheus Operator sets up metrics scraping]({{}}/img/rancher/set-up-scraping.svg) - -When the Prometheus Operator observes ServiceMonitors, PodMonitors and PrometheusRules being created, it knows that the scrape configuration needs to be updated in Prometheus. It updates Prometheus by first updating the configuration and rules files in the volumes of Prometheus's Deployment or StatefulSet. Then it calls the Prometheus API to sync the new configuration, resulting in the Prometheus Deployment or StatefulSet to be modified in place. - -![How the Prometheus Operator Updates Scrape Configuration]({{}}/img/rancher/update-scrape-config.svg) - -### 5.3. How Kubernetes Component Metrics are Exposed +### How Kubernetes Component Metrics are Exposed Prometheus scrapes metrics from deployments known as [exporters,](https://prometheus.io/docs/instrumenting/exporters/) which export the time series data in a format that Prometheus can ingest. In Prometheus, time series consist of streams of timestamped values belonging to the same metric and the same set of labeled dimensions. -To allow monitoring to be installed on hardened Kubernetes clusters, `rancher-monitoring` application proxies the communication between Prometheus and the exporter through PushProx for some Kubernetes master components. +### Scraping Metrics with PushProx -### 5.4. Scraping Metrics without PushProx +Certain internal Kubernetes components are scraped via a proxy deployed as part of Monitoring V2 called PushProx. For detailed information on PushProx, refer [here](#how-pushprox-works) and to the above [architecture](#1-architecture-overview) section. -The Kubernetes components that directly expose metrics to Prometheus are the following: +### Scraping Metrics -- kubelet -- ingress-nginx* +The following Kubernetes components are directly scraped by Prometheus: + +- kubelet* +- ingress-nginx** - coreDns/kubeDns - kube-api-server -\* For RKE and RKE2 clusters, ingress-nginx is deployed by default and treated as an internal Kubernetes component. +\* You can optionally use `hardenedKubelet.enabled` to use a PushProx, but that is not the default. -### 5.5. Scraping Metrics with PushProx +** For RKE and RKE2 clusters, ingress-nginx is deployed by default and treated as an internal Kubernetes component. -The purpose of this architecture is to allow us to scrape internal Kubernetes components without exposing those ports to inbound requests. As a result, Prometheus can scrape metrics across a network boundary. -The Kubernetes components that expose metrics to Prometheus through PushProx are the following: +### Scraping Metrics Based on Kubernetes Distribution -- kube-controller-manager -- kube-scheduler -- etcd -- kube-proxy - -For each PushProx exporter, we deploy one PushProx client onto all target nodes. For example, a PushProx client is deployed onto all controlplane nodes for kube-controller-manager, all etcd nodes for kube-etcd, and all nodes for kubelet. We deploy exactly one PushProx proxy per exporter. - -The process for exporting metrics is as follows: - -1. The PushProx Client establishes an outbound connection with the PushProx Proxy. -2. The client then polls the proxy for scrape requests that have come into the proxy. -3. When the proxy receives a scrape request from Prometheus, the client sees it as a result of the poll. -4. The client scrapes the internal component. -5. The internal component responds by pushing metrics back to the proxy. - -
Process for Exporting Metrics with PushProx
- -![Process for Exporting Metrics with PushProx]({{}}/img/rancher/pushprox-process.svg) - -Metrics are scraped differently based on the Kubernetes distribution. For help with terminology, see Terminology(#terminology). For details, see the table below: +Metrics are scraped differently based on the Kubernetes distribution. For help with terminology, refer [here](#terminology). For details, see the table below:
How Metrics are Exposed to Prometheus
@@ -239,7 +244,7 @@ Metrics are scraped differently based on the Kubernetes distribution. For help w \* For RKE and RKE2 clusters, ingress-nginx is deployed by default and treated as an internal Kubernetes component. -### 5.6. Terminology +### Terminology - **kube-scheduler:** The internal Kubernetes component that uses information in the pod spec to decide on which node to run a pod. - **kube-controller-manager:** The internal Kubernetes component that is responsible for node management (detecting if a node fails), pod replication and endpoint creation. diff --git a/content/rancher/v2.5/en/monitoring-alerting/rbac/_index.md b/content/rancher/v2.5/en/monitoring-alerting/rbac/_index.md index 4bc0e009c2a..3260cf8cf95 100644 --- a/content/rancher/v2.5/en/monitoring-alerting/rbac/_index.md +++ b/content/rancher/v2.5/en/monitoring-alerting/rbac/_index.md @@ -128,8 +128,8 @@ The relationship between the default roles deployed by Rancher Cluster Manager ( | --------- | --------- | --------- | --------- | | cluster-owner | cluster-admin | N/A | ClusterRoleBinding | | cluster-member | admin | monitoring-admin | ClusterRoleBinding | -| project-owner | edit | monitoring-admin | RoleBinding within Project namespace | -| project-member | view | monitoring-edit | RoleBinding within Project namespace | +| project-owner | admin | monitoring-admin | RoleBinding within Project namespace | +| project-member | edit | monitoring-edit | RoleBinding within Project namespace | In addition to these default Roles, the following additional Rancher project roles can be applied to members of your Cluster to provide additional access to Monitoring. These Rancher Roles will be tied to ClusterRoles deployed by the Monitoring chart: diff --git a/content/rancher/v2.5/en/overview/_index.md b/content/rancher/v2.5/en/overview/_index.md index dea44182edd..0bfde58d339 100644 --- a/content/rancher/v2.5/en/overview/_index.md +++ b/content/rancher/v2.5/en/overview/_index.md @@ -39,7 +39,7 @@ The Rancher API server is built on top of an embedded Kubernetes API server and - **Provisioning Kubernetes clusters:** The Rancher API server can [provision Kubernetes]({{}}/rancher/v2.5/en/cluster-provisioning/) on existing nodes, or perform [Kubernetes upgrades.]({{}}/rancher/v2.5/en/cluster-admin/upgrading-kubernetes) - **Catalog management:** Rancher provides the ability to use a [catalog of Helm charts]({{}}/rancher/v2.5/en/catalog/) that make it easy to repeatedly deploy applications. -- **Managing projects:** A project is a group of multiple namespaces and access control policies within a cluster. A project is a Rancher concept, not a Kubernetes concept, which allows you manage multiple namespaces as a group and perform Kubernetes operations in them. The Rancher UI provides features for [project administration]({{}}/rancher/v2.5/en/project-admin/) and for [managing applications within projects.]({{}}/rancher/v2.5/en/k8s-in-rancher/) +- **Managing projects:** A project is a group of multiple namespaces and access control policies within a cluster. A project is a Rancher concept, not a Kubernetes concept, which allows you to manage multiple namespaces as a group and perform Kubernetes operations in them. The Rancher UI provides features for [project administration]({{}}/rancher/v2.5/en/project-admin/) and for [managing applications within projects.]({{}}/rancher/v2.5/en/k8s-in-rancher/) - **Pipelines:** Setting up a [pipeline]({{}}/rancher/v2.5/en/project-admin/pipelines/) can help developers deliver new software as quickly and efficiently as possible. Within Rancher, you can configure pipelines for each of your Rancher projects. - **Istio:** Our [integration with Istio]({{}}/rancher/v2.5/en/istio/) is designed so that a Rancher operator, such as an administrator or cluster owner, can deliver Istio to developers. Then developers can use Istio to enforce security policies, troubleshoot problems, or manage traffic for green/blue deployments, canary deployments, or A/B testing. diff --git a/content/rancher/v2.5/en/overview/architecture-recommendations/_index.md b/content/rancher/v2.5/en/overview/architecture-recommendations/_index.md index a2aa6faf9f5..25029e83663 100644 --- a/content/rancher/v2.5/en/overview/architecture-recommendations/_index.md +++ b/content/rancher/v2.5/en/overview/architecture-recommendations/_index.md @@ -34,7 +34,7 @@ We don't recommend installing Rancher in a single Docker container, because if t ### K3s Kubernetes Cluster Installations -One option for the underlying Kubernetes cluster is to use K3s Kubernetes. K3s Rancher's CNCF certified Kubernetes distribution. It is easy to install, with half the memory of Kubernetes, all in a binary less than 100 MB. One main advantage of this K3s architecture is that it allows an external datastore to hold the cluster data, allowing the K3s server nodes to be treated as ephemeral. +One option for the underlying Kubernetes cluster is to use K3s Kubernetes. K3s is Rancher's CNCF certified Kubernetes distribution. It is easy to install and uses half the memory of Kubernetes, all in a binary of less than 100 MB. Another advantage of K3s is that it allows an external datastore to hold the cluster data, allowing the K3s server nodes to be treated as ephemeral.
Architecture of a K3s Kubernetes Cluster Running the Rancher Management Server
![Architecture of a K3s Kubernetes Cluster Running the Rancher Management Server]({{}}/img/rancher/k3s-server-storage.svg) @@ -110,4 +110,4 @@ For more best practices for downstream clusters, refer to the [production checkl If you are using an [authorized cluster endpoint,]({{}}/rancher/v2.5/en/overview/architecture/#4-authorized-cluster-endpoint) we recommend creating an FQDN pointing to a load balancer which balances traffic across your nodes with the `controlplane` role. -If you are using private CA signed certificates on the load balancer, you have to supply the CA certificate, which will be included in the generated kubeconfig file to validate the certificate chain. See the documentation on [kubeconfig files]({{}}/rancher/v2.5/en/k8s-in-rancher/kubeconfig/) and [API keys]({{}}/rancher/v2.5/en/user-settings/api-keys/#creating-an-api-key) for more information. \ No newline at end of file +If you are using private CA signed certificates on the load balancer, you have to supply the CA certificate, which will be included in the generated kubeconfig file to validate the certificate chain. See the documentation on [kubeconfig files]({{}}/rancher/v2.5/en/k8s-in-rancher/kubeconfig/) and [API keys]({{}}/rancher/v2.5/en/user-settings/api-keys/#creating-an-api-key) for more information. diff --git a/content/rancher/v2.5/en/overview/concepts/_index.md b/content/rancher/v2.5/en/overview/concepts/_index.md index 4e9d2d89e0d..30d5374875b 100644 --- a/content/rancher/v2.5/en/overview/concepts/_index.md +++ b/content/rancher/v2.5/en/overview/concepts/_index.md @@ -54,7 +54,7 @@ Three etcd nodes is generally sufficient for smaller clusters and five etcd node ### Controlplane Nodes -Controlplane nodes run the Kubernetes API server, scheduler, and controller manager. These nodes take care of routine tasks to ensure that your cluster maintains your configuration. Because all cluster data is stored on your etcd nodes, control plane nodes are stateless. You can run control plane on a single node, although two or more nodes are recommended for redundancy. Additionally, a single node can share the control plane and etcd roles. +Controlplane nodes run the Kubernetes API server, scheduler, and controller manager. These nodes take care of routine tasks to ensure that your cluster maintains your configuration. Because all cluster data is stored on your etcd nodes, control plane nodes are stateless. You can run control plane on a single node, although three or more nodes are recommended for redundancy. Additionally, a single node can share the control plane and etcd roles. ### Worker Nodes @@ -71,4 +71,4 @@ For high-availability installations of Rancher, Helm is the tool used to install Helm is the package management tool of choice for Kubernetes. Helm charts provide templating syntax for Kubernetes YAML manifest documents. With Helm we can create configurable deployments instead of just using static files. For more information about creating your own catalog of deployments, check out the docs at [https://helm.sh/](https://helm.sh). -For more information on service accounts and cluster role binding, refer to the [Kubernetes documentation.](https://kubernetes.io/docs/reference/access-authn-authz/rbac/) \ No newline at end of file +For more information on service accounts and cluster role binding, refer to the [Kubernetes documentation.](https://kubernetes.io/docs/reference/access-authn-authz/rbac/) diff --git a/content/rancher/v2.5/en/pipelines/_index.md b/content/rancher/v2.5/en/pipelines/_index.md index a47fdaf7577..8c7ac545a87 100644 --- a/content/rancher/v2.5/en/pipelines/_index.md +++ b/content/rancher/v2.5/en/pipelines/_index.md @@ -6,7 +6,9 @@ aliases: - /rancher/v2.x/en/pipelines/ --- -> As of Rancher v2.5, Git-based deployment pipelines are now recommended to be handled with Rancher Continuous Delivery powered by [Fleet,]({{}}/rancher/v2.5/en/deploy-across-clusters/fleet) available in Cluster Explorer. +> As of Rancher v2.5, Git-based deployment pipelines are now deprecated. We recommend handling pipelines with Rancher Continuous Delivery powered by [Fleet]({{}}/rancher/v2.5/en/deploy-across-clusters/fleet), available in Cluster Explorer. +> +>**Notice:** Fleet does not replace Rancher pipelines; the distinction is that Rancher pipelines are now powered by Fleet. Rancher's pipeline provides a simple CI/CD experience. Use it to automatically checkout code, run builds or scripts, publish Docker images or catalog applications, and deploy the updated software to users. diff --git a/content/rancher/v2.5/en/project-admin/project-members/_index.md b/content/rancher/v2.5/en/project-admin/project-members/_index.md index a8476db8ec8..45a149e90ea 100644 --- a/content/rancher/v2.5/en/project-admin/project-members/_index.md +++ b/content/rancher/v2.5/en/project-admin/project-members/_index.md @@ -43,6 +43,8 @@ Following project creation, you can add users as project members so that they ca > >- Users assigned the `Owner` or `Member` role for a project automatically inherit the `namespace creation` role. However, this role is a [Kubernetes ClusterRole](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#role-and-clusterrole), meaning its scope extends to all projects in the cluster. Therefore, users explicitly assigned the `Owner` or `Member` role for a project can create namespaces in other projects they're assigned to, even with only the `Read Only` role assigned. > + >- By default, the Rancher role of `project-member` inherits from the `Kubernetes-edit` role, and the `project-owner` role inherits from the `Kubernetes-admin` role. As such, both `project-member` and `project-owner` roles will allow for namespace management, including the ability to create and delete namespaces. + > >- For `Custom` roles, you can modify the list of individual roles available for assignment. > > - To add roles to the list, [Add a Custom Role]({{}}/rancher/v2.5/en/admin-settings/rbac/default-custom-roles). diff --git a/content/rancher/v2.5/en/project-admin/resource-quotas/_index.md b/content/rancher/v2.5/en/project-admin/resource-quotas/_index.md index 77f54f689cc..5bd7f0ae075 100644 --- a/content/rancher/v2.5/en/project-admin/resource-quotas/_index.md +++ b/content/rancher/v2.5/en/project-admin/resource-quotas/_index.md @@ -42,4 +42,4 @@ Edit [resource quotas]({{}}/rancher/v2.5/en/cluster-admin/projects-and- 1. Click **Create**. -**Result:** The resource quota is applied to your project and namespaces. When you add more namespaces in the future, Rancher validates that the project can accommodate the namespace. If the project can't allocate the resources, Rancher won't let you save your changes. +**Result:** The resource quota is applied to your project and namespaces. When you add more namespaces in the future, Rancher validates that the project can accommodate the namespace. If the project can't allocate the resources, you may still create namespaces, but they will be given a resource quota of 0. Subsequently, Rancher will not allow you to create any resources restricted by this quota. diff --git a/content/rancher/v2.5/en/quick-start-guide/_index.md b/content/rancher/v2.5/en/quick-start-guide/_index.md index 13ea050b05e..6e8bb1f38ee 100644 --- a/content/rancher/v2.5/en/quick-start-guide/_index.md +++ b/content/rancher/v2.5/en/quick-start-guide/_index.md @@ -1,10 +1,8 @@ --- title: Rancher Deployment Quick Start Guides metaDescription: Use this section to jump start your Rancher deployment and testing. It contains instructions for a simple Rancher setup and some common use cases. -short title: Use this section to jump start your Rancher deployment and testing. It contains instructions for a simple Rancher setup and some common use cases. +short title: Use this section to jump start your Rancher deployment and testing. It contains instructions for a simple Rancher setup and some common use cases. weight: 2 -aliases: - - /rancher/v2.x/en/quick-start-guide/ --- >**Note:** The intent of these guides is to quickly launch a sandbox that you can use to evaluate Rancher. These guides are not intended for production environments. For comprehensive setup instructions, see [Installation]({{}}/rancher/v2.5/en/installation/). @@ -15,5 +13,3 @@ We have Quick Start Guides for: - [Deploying Rancher Server]({{}}/rancher/v2.5/en/quick-start-guide/deployment/): Get started running Rancher using the method most convenient for you. - [Deploying Workloads]({{}}/rancher/v2.5/en/quick-start-guide/workload/): Deploy a simple [workload](https://kubernetes.io/docs/concepts/workloads/) and expose it, letting you access it from outside the cluster. - -- [Using the CLI]({{}}/rancher/v2.5/en/quick-start-guide/cli/): Use `kubectl` or Rancher command line interface (CLI) to interact with your Rancher instance. diff --git a/content/rancher/v2.5/en/quick-start-guide/cli/_index.md b/content/rancher/v2.5/en/quick-start-guide/cli/_index.md deleted file mode 100644 index 954e27f6a70..00000000000 --- a/content/rancher/v2.5/en/quick-start-guide/cli/_index.md +++ /dev/null @@ -1,41 +0,0 @@ ---- -title: CLI with Rancher -weight: 100 -aliases: - - /rancher/v2.x/en/quick-start-guide/cli/ ---- - -Interact with Rancher using command line interface (CLI) tools from your workstation. - -## Rancher CLI - -Follow the steps in [rancher cli](../../cli). - -Ensure you can run `rancher kubectl get pods` successfully. - - -## kubectl -Install the `kubectl` utility. See [install kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/). - - -Configure kubectl by visiting your cluster in the Rancher Web UI then clicking on `Kubeconfig`, copying contents and putting into your `~/.kube/config` file. - -Run `kubectl cluster-info` or `kubectl get pods` successfully. - -## Authentication with kubectl and kubeconfig Tokens with TTL - -_Requirements_ - -If admins have [enforced TTL on kubeconfig tokens]({{}}/rancher/v2.5/en/api/api-tokens/#setting-ttl-on-kubeconfig-tokens), the kubeconfig file requires the [Rancher cli](../cli) to be present in your PATH when you run `kubectl`. Otherwise, you’ll see error like: -`Unable to connect to the server: getting credentials: exec: exec: "rancher": executable file not found in $PATH`. - -This feature enables kubectl to authenticate with the Rancher server and get a new kubeconfig token when required. The following auth providers are currently supported: - -1. Local -2. Active Directory -3. FreeIpa, OpenLdap -4. SAML providers - Ping, Okta, ADFS, Keycloak, Shibboleth - -When you first run kubectl, for example, `kubectl get pods`, it will ask you to pick an auth provider and log in with the Rancher server. -The kubeconfig token is cached in the path where you run kubectl under `./.cache/token`. This token is valid till [it expires](../../api/api-tokens/#setting-ttl-on-kubeconfig-tokens-period), or [gets deleted from the Rancher server](../../api/api-tokens/#deleting-tokens) -Upon expiration, the next `kubectl get pods` will ask you to log in with the Rancher server again. diff --git a/content/rancher/v2.5/en/quick-start-guide/deployment/amazon-aws-qs/_index.md b/content/rancher/v2.5/en/quick-start-guide/deployment/amazon-aws-qs/_index.md index c5f189a9542..146b7151ad4 100644 --- a/content/rancher/v2.5/en/quick-start-guide/deployment/amazon-aws-qs/_index.md +++ b/content/rancher/v2.5/en/quick-start-guide/deployment/amazon-aws-qs/_index.md @@ -1,11 +1,11 @@ --- title: Rancher AWS Quick Start Guide -description: Read this step by step Rancher AWS guide to quickly deploy a Rancher Server with a single node cluster attached. +description: Read this step by step Rancher AWS guide to quickly deploy a Rancher server with a single-node downstream Kubernetes cluster attached. weight: 100 -aliases: - - /rancher/v2.x/en/quick-start-guide/deployment/amazon-aws-qs/ --- -The following steps will quickly deploy a Rancher Server on AWS with a single node cluster attached. +The following steps will quickly deploy a Rancher server on AWS in a single-node K3s Kubernetes cluster, with a single-node downstream Kubernetes cluster attached. + +>**Note:** The intent of these guides is to quickly launch a sandbox that you can use to evaluate Rancher. These guides are not intended for production environments. For comprehensive setup instructions, see [Installation]({{}}/rancher/v2.5/en/installation/). ## Prerequisites @@ -21,25 +21,26 @@ The following steps will quickly deploy a Rancher Server on AWS with a single no 1. Clone [Rancher Quickstart](https://github.com/rancher/quickstart) to a folder using `git clone https://github.com/rancher/quickstart`. -1. Go into the AWS folder containing the terraform files by executing `cd quickstart/aws`. +2. Go into the AWS folder containing the terraform files by executing `cd quickstart/aws`. -1. Rename the `terraform.tfvars.example` file to `terraform.tfvars`. +3. Rename the `terraform.tfvars.example` file to `terraform.tfvars`. -1. Edit `terraform.tfvars` and customize the following variables: +4. Edit `terraform.tfvars` and customize the following variables: - `aws_access_key` - Amazon AWS Access Key - `aws_secret_key` - Amazon AWS Secret Key - `rancher_server_admin_password` - Admin password for created Rancher server -1. **Optional:** Modify optional variables within `terraform.tfvars`. +5. **Optional:** Modify optional variables within `terraform.tfvars`. See the [Quickstart Readme](https://github.com/rancher/quickstart) and the [AWS Quickstart Readme](https://github.com/rancher/quickstart/tree/master/aws) for more information. Suggestions include: - - `aws_region` - Amazon AWS region, choose the closest instead of the default + - `aws_region` - Amazon AWS region, choose the closest instead of the default (`us-east-1`) - `prefix` - Prefix for all created resources - `instance_type` - EC2 instance size used, minimum is `t3a.medium` but `t3a.large` or `t3a.xlarge` could be used if within budget + - `add_windows_node` - If true, an additional Windows worker node is added to the workload cluster -1. Run `terraform init`. +6. Run `terraform init`. -1. To initiate the creation of the environment, run `terraform apply --auto-approve`. Then wait for output similar to the following: +7. To initiate the creation of the environment, run `terraform apply --auto-approve`. Then wait for output similar to the following: ``` Apply complete! Resources: 16 added, 0 changed, 0 destroyed. @@ -51,11 +52,12 @@ Suggestions include: workload_node_ip = yy.yy.yy.yy ``` -1. Paste the `rancher_server_url` from the output above into the browser. Log in when prompted (default username is `admin`, use the password set in `rancher_server_admin_password`). +8. Paste the `rancher_server_url` from the output above into the browser. Log in when prompted (default username is `admin`, use the password set in `rancher_server_admin_password`). +9. ssh to the Rancher server using the `id_rsa` key generated in `quickstart/aws`. #### Result -Two Kubernetes clusters are deployed into your AWS account, one running Rancher Server and the other ready for experimentation deployments. Please note that while this setup is a great way to explore Rancher functionality, a production setup should follow our high availability setup guidelines. +Two Kubernetes clusters are deployed into your AWS account, one running Rancher Server and the other ready for experimentation deployments. Please note that while this setup is a great way to explore Rancher functionality, a production setup should follow our high availability setup guidelines. SSH keys for the VMs are auto-generated and stored in the module directory. ### What's Next? diff --git a/content/rancher/v2.5/en/quick-start-guide/deployment/digital-ocean-qs/_index.md b/content/rancher/v2.5/en/quick-start-guide/deployment/digital-ocean-qs/_index.md index 2e3e94777df..168b580dbdf 100644 --- a/content/rancher/v2.5/en/quick-start-guide/deployment/digital-ocean-qs/_index.md +++ b/content/rancher/v2.5/en/quick-start-guide/deployment/digital-ocean-qs/_index.md @@ -1,11 +1,11 @@ --- title: Rancher DigitalOcean Quick Start Guide -description: Read this step by step Rancher DigitalOcean guide to quickly deploy a Rancher Server with a single node cluster attached. +description: Read this step by step Rancher DigitalOcean guide to quickly deploy a Rancher server with a single-node downstream Kubernetes cluster attached. weight: 100 -aliases: - - /rancher/v2.x/en/quick-start-guide/deployment/digital-ocean-qs/ --- -The following steps will quickly deploy a Rancher Server on DigitalOcean with a single node cluster attached. +The following steps will quickly deploy a Rancher server on DigitalOcean in a single-node K3s Kubernetes cluster, with a single-node downstream Kubernetes cluster attached. + +>**Note:** The intent of these guides is to quickly launch a sandbox that you can use to evaluate Rancher. These guides are not intended for production environments. For comprehensive setup instructions, see [Installation]({{}}/rancher/v2.5/en/installation/). ## Prerequisites @@ -21,25 +21,24 @@ The following steps will quickly deploy a Rancher Server on DigitalOcean with a 1. Clone [Rancher Quickstart](https://github.com/rancher/quickstart) to a folder using `git clone https://github.com/rancher/quickstart`. -1. Go into the DigitalOcean folder containing the terraform files by executing `cd quickstart/do`. +2. Go into the DigitalOcean folder containing the terraform files by executing `cd quickstart/do`. -1. Rename the `terraform.tfvars.example` file to `terraform.tfvars`. +3. Rename the `terraform.tfvars.example` file to `terraform.tfvars`. -1. Edit `terraform.tfvars` and customize the following variables: +4. Edit `terraform.tfvars` and customize the following variables: - `do_token` - DigitalOcean access key - `rancher_server_admin_password` - Admin password for created Rancher server -1. **Optional:** Modify optional variables within `terraform.tfvars`. +5. **Optional:** Modify optional variables within `terraform.tfvars`. See the [Quickstart Readme](https://github.com/rancher/quickstart) and the [DO Quickstart Readme](https://github.com/rancher/quickstart/tree/master/do) for more information. Suggestions include: - - `do_region` - DigitalOcean region, choose the closest instead of the default + - `do_region` - DigitalOcean region, choose the closest instead of the default (`nyc1`) - `prefix` - Prefix for all created resources - `droplet_size` - Droplet size used, minimum is `s-2vcpu-4gb` but `s-4vcpu-8gb` could be used if within budget - - `ssh_key_file_name` - Use a specific SSH key instead of `~/.ssh/id_rsa` (public key is assumed to be `${ssh_key_file_name}.pub`) -1. Run `terraform init`. +6. Run `terraform init`. -1. To initiate the creation of the environment, run `terraform apply --auto-approve`. Then wait for output similar to the following: +7. To initiate the creation of the environment, run `terraform apply --auto-approve`. Then wait for output similar to the following: ``` Apply complete! Resources: 15 added, 0 changed, 0 destroyed. @@ -51,11 +50,12 @@ Suggestions include: workload_node_ip = yy.yy.yy.yy ``` -1. Paste the `rancher_server_url` from the output above into the browser. Log in when prompted (default username is `admin`, use the password set in `rancher_server_admin_password`). +8. Paste the `rancher_server_url` from the output above into the browser. Log in when prompted (default username is `admin`, use the password set in `rancher_server_admin_password`). +9. ssh to the Rancher Server using the `id_rsa` key generated in `quickstart/do`. #### Result -Two Kubernetes clusters are deployed into your DigitalOcean account, one running Rancher Server and the other ready for experimentation deployments. +Two Kubernetes clusters are deployed into your DigitalOcean account, one running Rancher Server and the other ready for experimentation deployments. Please note that while this setup is a great way to explore Rancher functionality, a production setup should follow our high availability setup guidelines. SSH keys for the VMs are auto-generated and stored in the module directory. ### What's Next? diff --git a/content/rancher/v2.5/en/quick-start-guide/deployment/google-gcp-qs/_index.md b/content/rancher/v2.5/en/quick-start-guide/deployment/google-gcp-qs/_index.md index 48eaec3b4ba..35839e68f87 100644 --- a/content/rancher/v2.5/en/quick-start-guide/deployment/google-gcp-qs/_index.md +++ b/content/rancher/v2.5/en/quick-start-guide/deployment/google-gcp-qs/_index.md @@ -1,69 +1,70 @@ ---- -title: Rancher GCP Quick Start Guide -description: Read this step by step Rancher GCP guide to quickly deploy a Rancher Server with a single node cluster attached. -weight: 100 -aliases: - - /rancher/v2.x/en/quick-start-guide/deployment/google-gcp-qs/ ---- -The following steps will quickly deploy a Rancher server on GCP in a single-node RKE Kubernetes cluster, with a single-node downstream Kubernetes cluster attached. - -## Prerequisites - ->**Note** ->Deploying to Google GCP will incur charges. - -- [Google GCP Account](https://console.cloud.google.com/): A Google GCP Account is required to create resources for deploying Rancher and Kubernetes. -- [Google GCP Project](https://cloud.google.com/appengine/docs/standard/nodejs/building-app/creating-project): Use this link to follow a tutorial to create a GCP Project if you don't have one yet. -- [Google GCP Service Account](https://cloud.google.com/iam/docs/creating-managing-service-account-keys): Use this link and follow instructions to create a GCP service account and token file. -- [Terraform](https://www.terraform.io/downloads.html): Used to provision the server and cluster in Google GCP. - - -## Getting Started - -1. Clone [Rancher Quickstart](https://github.com/rancher/quickstart) to a folder using `git clone https://github.com/rancher/quickstart`. - -1. Go into the GCP folder containing the terraform files by executing `cd quickstart/gcp`. - -1. Rename the `terraform.tfvars.example` file to `terraform.tfvars`. - -1. Edit `terraform.tfvars` and customize the following variables: - - `gcp_account_json` - GCP service account file path and file name - - `rancher_server_admin_password` - Admin password for created Rancher server - -1. **Optional:** Modify optional variables within `terraform.tfvars`. -See the [Quickstart Readme](https://github.com/rancher/quickstart) and the [GCP Quickstart Readme](https://github.com/rancher/quickstart/tree/master/gcp) for more information. -Suggestions include: - - `gcp_region` - Google GCP region, choose the closest instead of the default - - `prefix` - Prefix for all created resources - - `machine_type` - Compute instance size used, minimum is `n1-standard-1` but `n1-standard-2` or `n1-standard-4` could be used if within budget - - `ssh_key_file_name` - Use a specific SSH key instead of `~/.ssh/id_rsa` (public key is assumed to be `${ssh_key_file_name}.pub`) - -1. Run `terraform init`. - -1. To initiate the creation of the environment, run `terraform apply --auto-approve`. Then wait for output similar to the following: - - ``` - Apply complete! Resources: 16 added, 0 changed, 0 destroyed. - - Outputs: - - rancher_node_ip = xx.xx.xx.xx - rancher_server_url = https://rancher.xx.xx.xx.xx.sslip.io - workload_node_ip = yy.yy.yy.yy - ``` - -1. Paste the `rancher_server_url` from the output above into the browser. Log in when prompted (default username is `admin`, use the password set in `rancher_server_admin_password`). - -#### Result - -Two Kubernetes clusters are deployed into your GCP account, one running Rancher Server and the other ready for experimentation deployments. - -### What's Next? - -Use Rancher to create a deployment. For more information, see [Creating Deployments]({{< baseurl >}}/rancher/v2.5/en/quick-start-guide/workload). - -## Destroying the Environment - -1. From the `quickstart/gcp` folder, execute `terraform destroy --auto-approve`. - -2. Wait for confirmation that all resources have been destroyed. +--- +title: Rancher GCP Quick Start Guide +description: Read this step by step Rancher GCP guide to quickly deploy a Rancher server with a single-node downstream Kubernetes cluster attached. +weight: 100 +--- +The following steps will quickly deploy a Rancher server on GCP in a single-node K3s Kubernetes cluster, with a single-node downstream Kubernetes cluster attached. + +>**Note:** The intent of these guides is to quickly launch a sandbox that you can use to evaluate Rancher. These guides are not intended for production environments. For comprehensive setup instructions, see [Installation]({{}}/rancher/v2.5/en/installation/). + +## Prerequisites + +>**Note** +>Deploying to Google GCP will incur charges. + +- [Google GCP Account](https://console.cloud.google.com/): A Google GCP Account is required to create resources for deploying Rancher and Kubernetes. +- [Google GCP Project](https://cloud.google.com/appengine/docs/standard/nodejs/building-app/creating-project): Use this link to follow a tutorial to create a GCP Project if you don't have one yet. +- [Google GCP Service Account](https://cloud.google.com/iam/docs/creating-managing-service-account-keys): Use this link and follow instructions to create a GCP service account and token file. +- [Terraform](https://www.terraform.io/downloads.html): Used to provision the server and cluster in Google GCP. + + +## Getting Started + +1. Clone [Rancher Quickstart](https://github.com/rancher/quickstart) to a folder using `git clone https://github.com/rancher/quickstart`. + +2. Go into the GCP folder containing the terraform files by executing `cd quickstart/gcp`. + +3. Rename the `terraform.tfvars.example` file to `terraform.tfvars`. + +4. Edit `terraform.tfvars` and customize the following variables: + - `gcp_account_json` - GCP service account file path and file name + - `rancher_server_admin_password` - Admin password for created Rancher server + +5. **Optional:** Modify optional variables within `terraform.tfvars`. +See the [Quickstart Readme](https://github.com/rancher/quickstart) and the [GCP Quickstart Readme](https://github.com/rancher/quickstart/tree/master/gcp) for more information. +Suggestions include: + - `gcp_region` - Google GCP region, choose the closest instead of the default (`us-east4`) + - `gcp_zone` - Google GCP zone, choose the closest instead of the default (`us-east4-a`) + - `prefix` - Prefix for all created resources + - `machine_type` - Compute instance size used, minimum is `n1-standard-1` but `n1-standard-2` or `n1-standard-4` could be used if within budget + +6. Run `terraform init`. + +7. To initiate the creation of the environment, run `terraform apply --auto-approve`. Then wait for output similar to the following: + + ``` + Apply complete! Resources: 16 added, 0 changed, 0 destroyed. + + Outputs: + + rancher_node_ip = xx.xx.xx.xx + rancher_server_url = https://rancher.xx.xx.xx.xx.sslip.io + workload_node_ip = yy.yy.yy.yy + ``` + +8. Paste the `rancher_server_url` from the output above into the browser. Log in when prompted (default username is `admin`, use the password set in `rancher_server_admin_password`). +9. ssh to the Rancher Server using the `id_rsa` key generated in `quickstart/gcp`. + +#### Result + +Two Kubernetes clusters are deployed into your GCP account, one running Rancher Server and the other ready for experimentation deployments. Please note that while this setup is a great way to explore Rancher functionality, a production setup should follow our high availability setup guidelines. SSH keys for the VMs are auto-generated and stored in the module directory. + +### What's Next? + +Use Rancher to create a deployment. For more information, see [Creating Deployments]({{< baseurl >}}/rancher/v2.5/en/quick-start-guide/workload). + +## Destroying the Environment + +1. From the `quickstart/gcp` folder, execute `terraform destroy --auto-approve`. + +2. Wait for confirmation that all resources have been destroyed. diff --git a/content/rancher/v2.5/en/quick-start-guide/deployment/microsoft-azure-qs/_index.md b/content/rancher/v2.5/en/quick-start-guide/deployment/microsoft-azure-qs/_index.md index a77fede6147..72c5d43096e 100644 --- a/content/rancher/v2.5/en/quick-start-guide/deployment/microsoft-azure-qs/_index.md +++ b/content/rancher/v2.5/en/quick-start-guide/deployment/microsoft-azure-qs/_index.md @@ -1,73 +1,76 @@ ---- -title: Rancher Azure Quick Start Guide -description: Read this step by step Rancher Azure guide to quickly deploy a Rancher Server with a single node cluster attached. -weight: 100 -aliases: - - /rancher/v2.x/en/quick-start-guide/deployment/microsoft-azure-qs/ ---- - -The following steps will quickly deploy a Rancher server on Azure in a single-node RKE Kubernetes cluster, with a single-node downstream Kubernetes cluster attached. - -## Prerequisites - ->**Note** ->Deploying to Microsoft Azure will incur charges. - -- [Microsoft Azure Account](https://azure.microsoft.com/en-us/free/): A Microsoft Azure Account is required to create resources for deploying Rancher and Kubernetes. -- [Microsoft Azure Subscription](https://docs.microsoft.com/en-us/azure/cost-management-billing/manage/create-subscription#create-a-subscription-in-the-azure-portal): Use this link to follow a tutorial to create a Microsoft Azure subscription if you don't have one yet. -- [Micsoroft Azure Tenant](https://docs.microsoft.com/en-us/azure/active-directory/develop/quickstart-create-new-tenant): Use this link and follow instructions to create a Microsoft Azure tenant. -- [Microsoft Azure Client ID/Secret](https://docs.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal): Use this link and follow instructions to create a Microsoft Azure client and secret. -- [Terraform](https://www.terraform.io/downloads.html): Used to provision the server and cluster in Microsoft Azure. - - -## Getting Started - -1. Clone [Rancher Quickstart](https://github.com/rancher/quickstart) to a folder using `git clone https://github.com/rancher/quickstart`. - -1. Go into the Azure folder containing the terraform files by executing `cd quickstart/azure`. - -1. Rename the `terraform.tfvars.example` file to `terraform.tfvars`. - -1. Edit `terraform.tfvars` and customize the following variables: - - `azure_subscription_id` - Microsoft Azure Subscription ID - - `azure_client_id` - Microsoft Azure Client ID - - `azure_client_secret` - Microsoft Azure Client Secret - - `azure_tenant_id` - Microsoft Azure Tenant ID - - `rancher_server_admin_password` - Admin password for created Rancher server - -2. **Optional:** Modify optional variables within `terraform.tfvars`. -See the [Quickstart Readme](https://github.com/rancher/quickstart) and the [Azure Quickstart Readme](https://github.com/rancher/quickstart/tree/master/azure) for more information. -Suggestions include: - - `azure_location` - Microsoft Azure region, choose the closest instead of the default - - `prefix` - Prefix for all created resources - - `instance_type` - Compute instance size used, minimum is `Standard_DS2_v2` but `Standard_DS2_v3` or `Standard_DS3_v2` could be used if within budget - -1. Run `terraform init`. - -1. To initiate the creation of the environment, run `terraform apply --auto-approve`. Then wait for output similar to the following: - - ``` - Apply complete! Resources: 16 added, 0 changed, 0 destroyed. - - Outputs: - - rancher_node_ip = xx.xx.xx.xx - rancher_server_url = https://rancher.xx.xx.xx.xx.sslip.io - workload_node_ip = yy.yy.yy.yy - ``` - -1. Paste the `rancher_server_url` from the output above into the browser. Log in when prompted (default username is `admin`, use the password set in `rancher_server_admin_password`). -2. ssh to the Rancher Server using the `id_rsa` key generated in `quickstart/azure`. -#### Result - -Two Kubernetes clusters are deployed into your Azure account, one running Rancher Server and the other ready for experimentation deployments. - -### What's Next? - -Use Rancher to create a deployment. For more information, see [Creating Deployments]({{< baseurl >}}/rancher/v2.5/en/quick-start-guide/workload). - -## Destroying the Environment - -1. From the `quickstart/azure` folder, execute `terraform destroy --auto-approve`. - -2. Wait for confirmation that all resources have been destroyed. +--- +title: Rancher Azure Quick Start Guide +description: Read this step by step Rancher Azure guide to quickly deploy a Rancher server with a single-node downstream Kubernetes cluster attached. +weight: 100 +--- + +The following steps will quickly deploy a Rancher server on Azure in a single-node K3s Kubernetes cluster, with a single-node downstream Kubernetes cluster attached. + +>**Note:** The intent of these guides is to quickly launch a sandbox that you can use to evaluate Rancher. These guides are not intended for production environments. For comprehensive setup instructions, see [Installation]({{}}/rancher/v2.5/en/installation/). + +## Prerequisites + +>**Note** +>Deploying to Microsoft Azure will incur charges. + +- [Microsoft Azure Account](https://azure.microsoft.com/en-us/free/): A Microsoft Azure Account is required to create resources for deploying Rancher and Kubernetes. +- [Microsoft Azure Subscription](https://docs.microsoft.com/en-us/azure/cost-management-billing/manage/create-subscription#create-a-subscription-in-the-azure-portal): Use this link to follow a tutorial to create a Microsoft Azure subscription if you don't have one yet. +- [Micsoroft Azure Tenant](https://docs.microsoft.com/en-us/azure/active-directory/develop/quickstart-create-new-tenant): Use this link and follow instructions to create a Microsoft Azure tenant. +- [Microsoft Azure Client ID/Secret](https://docs.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal): Use this link and follow instructions to create a Microsoft Azure client and secret. +- [Terraform](https://www.terraform.io/downloads.html): Used to provision the server and cluster in Microsoft Azure. + + +## Getting Started + +1. Clone [Rancher Quickstart](https://github.com/rancher/quickstart) to a folder using `git clone https://github.com/rancher/quickstart`. + +2. Go into the Azure folder containing the terraform files by executing `cd quickstart/azure`. + +3. Rename the `terraform.tfvars.example` file to `terraform.tfvars`. + +4. Edit `terraform.tfvars` and customize the following variables: + - `azure_subscription_id` - Microsoft Azure Subscription ID + - `azure_client_id` - Microsoft Azure Client ID + - `azure_client_secret` - Microsoft Azure Client Secret + - `azure_tenant_id` - Microsoft Azure Tenant ID + - `rancher_server_admin_password` - Admin password for created Rancher server + +5. **Optional:** Modify optional variables within `terraform.tfvars`. +See the [Quickstart Readme](https://github.com/rancher/quickstart) and the [Azure Quickstart Readme](https://github.com/rancher/quickstart/tree/master/azure) for more information. +Suggestions include: + - `azure_location` - Microsoft Azure region, choose the closest instead of the default (`East US`) + - `prefix` - Prefix for all created resources + - `instance_type` - Compute instance size used, minimum is `Standard_DS2_v2` but `Standard_DS2_v3` or `Standard_DS3_v2` could be used if within budget + - `add_windows_node` - If true, an additional Windows worker node is added to the workload cluster + - `windows_admin_password` - The admin password of the windows worker node + +6. Run `terraform init`. + +7. To initiate the creation of the environment, run `terraform apply --auto-approve`. Then wait for output similar to the following: + + ``` + Apply complete! Resources: 16 added, 0 changed, 0 destroyed. + + Outputs: + + rancher_node_ip = xx.xx.xx.xx + rancher_server_url = https://rancher.xx.xx.xx.xx.sslip.io + workload_node_ip = yy.yy.yy.yy + ``` + +8. Paste the `rancher_server_url` from the output above into the browser. Log in when prompted (default username is `admin`, use the password set in `rancher_server_admin_password`). +9. ssh to the Rancher Server using the `id_rsa` key generated in `quickstart/azure`. + +#### Result + +Two Kubernetes clusters are deployed into your Azure account, one running Rancher Server and the other ready for experimentation deployments. Please note that while this setup is a great way to explore Rancher functionality, a production setup should follow our high availability setup guidelines. SSH keys for the VMs are auto-generated and stored in the module directory. + +### What's Next? + +Use Rancher to create a deployment. For more information, see [Creating Deployments]({{< baseurl >}}/rancher/v2.5/en/quick-start-guide/workload). + +## Destroying the Environment + +1. From the `quickstart/azure` folder, execute `terraform destroy --auto-approve`. + +2. Wait for confirmation that all resources have been destroyed. diff --git a/content/rancher/v2.5/en/quick-start-guide/deployment/quickstart-manual-setup/_index.md b/content/rancher/v2.5/en/quick-start-guide/deployment/quickstart-manual-setup/_index.md index 770940ff92f..e9051c73081 100644 --- a/content/rancher/v2.5/en/quick-start-guide/deployment/quickstart-manual-setup/_index.md +++ b/content/rancher/v2.5/en/quick-start-guide/deployment/quickstart-manual-setup/_index.md @@ -10,6 +10,8 @@ Howdy Partner! This tutorial walks you through: - Creation of your first cluster - Deployment of an application, Nginx +>**Note:** The intent of these guides is to quickly launch a sandbox that you can use to evaluate Rancher. These guides are not intended for production environments. For comprehensive setup instructions, see [Installation]({{}}/rancher/v2.5/en/installation/). + ## Quick Start Outline This Quick Start Guide is divided into different tasks for easier consumption. @@ -48,7 +50,7 @@ To install Rancher on your host, connect to it and then use a shell to install. 1. Log in to your Linux host using your preferred shell, such as PuTTy or a remote Terminal connection. -2. From your shell, enter the following command: +1. From your shell, enter the following command: ``` sudo docker run -d --restart=unless-stopped -p 80:80 -p 443:443 --privileged rancher/rancher @@ -60,13 +62,17 @@ To install Rancher on your host, connect to it and then use a shell to install. Log in to Rancher to begin using the application. After you log in, you'll make some one-time configurations. -1. Open a web browser and enter the IP address of your host: `https://`. +1. Open a web browser and enter the IP address of your host: `https://`. Replace `` with your host IP address. -2. When prompted, create a password for the default `admin` account there cowpoke! +1. When prompted, create a password for the default `admin` account there cowpoke! -3. Set the **Rancher Server URL**. The URL can either be an IP address or a host name. However, each node added to your cluster must be able to connect to this URL.

If you use a hostname in the URL, this hostname must be resolvable by DNS on the nodes you want to add to you cluster. +1. Set the **Default View**. + - If `I want to create or manage multiple clusters` is selected, the Cluster Manager UI is used as the default view. + - If `I'm only going to use the cluster Rancher was installed on` is selected, the Cluster Explorer UI is used as the default view. + +1. Set the **Rancher Server URL**. The URL can either be an IP address or a host name. However, each node added to your cluster must be able to connect to this URL.

If you use a hostname in the URL, this hostname must be resolvable by DNS on the nodes you want to add to you cluster.
@@ -76,35 +82,37 @@ Welcome to Rancher! You are now able to create your first Kubernetes cluster. In this task, you can use the versatile **Custom** option. This option lets you add _any_ Linux host (cloud-hosted VM, on-prem VM, or bare-metal) to be used in a cluster. +1. If you chose `I'm only going to use the cluster Rancher was installed on` when setting the default view, click the **Cluster Manager** button in the upper-right of the UI to access the **Clusters** page. + 1. From the **Clusters** page, click **Add Cluster**. -2. Choose **Existing Nodes**. +1. Choose **Existing Nodes**. -3. Enter a **Cluster Name**. +1. Enter a **Cluster Name**. -4. Skip **Member Roles** and **Cluster Options**. We'll tell you about them later. +1. Skip **Member Roles** and **Cluster Options**. We'll tell you about them later. -5. Click **Next**. +1. Click **Next**. -6. From **Node Role**, select _all_ the roles: **etcd**, **Control**, and **Worker**. +1. From **Node Role**, select _all_ the roles: **etcd**, **Control**, and **Worker**. -7. **Optional**: Rancher auto-detects the IP addresses used for Rancher communication and cluster communication. You can override these using `Public Address` and `Internal Address` in the **Node Address** section. +1. **Optional**: Rancher auto-detects the IP addresses used for Rancher communication and cluster communication. You can override these using `Public Address` and `Internal Address` in the **Node Address** section. -8. Skip the **Labels** stuff. It's not important for now. +1. Skip the **Labels** stuff. It's not important for now. -9. Copy the command displayed on screen to your clipboard. +1. Copy the command displayed on screen to your clipboard. -10. Log in to your Linux host using your preferred shell, such as PuTTy or a remote Terminal connection. Run the command copied to your clipboard. +1. Log in to your Linux host using your preferred shell, such as PuTTy or a remote Terminal connection. Run the command copied to your clipboard. -11. When you finish running the command on your Linux host, click **Done**. +1. When you finish running the command on your Linux host, click **Done**. -**Result:** +**Result:** Your cluster is created and assigned a state of **Provisioning.** Rancher is standing up your cluster. You can access your cluster after its state is updated to **Active.** -**Active** clusters are assigned two Projects: +**Active** clusters are assigned two Projects: - `Default`, containing the `default` namespace - `System`, containing the `cattle-system`, `ingress-nginx`, `kube-public`, and `kube-system` namespaces diff --git a/content/rancher/v2.5/en/quick-start-guide/deployment/quickstart-vagrant/_index.md b/content/rancher/v2.5/en/quick-start-guide/deployment/quickstart-vagrant/_index.md index b175645408d..43102865117 100644 --- a/content/rancher/v2.5/en/quick-start-guide/deployment/quickstart-vagrant/_index.md +++ b/content/rancher/v2.5/en/quick-start-guide/deployment/quickstart-vagrant/_index.md @@ -6,6 +6,8 @@ aliases: --- The following steps quickly deploy a Rancher Server with a single node cluster attached. +>**Note:** The intent of these guides is to quickly launch a sandbox that you can use to evaluate Rancher. These guides are not intended for production environments. For comprehensive setup instructions, see [Installation]({{}}/rancher/v2.5/en/installation/). + ## Prerequisites - [Vagrant](https://www.vagrantup.com): Vagrant is required as this is used to provision the machine based on the Vagrantfile. @@ -16,7 +18,7 @@ The following steps quickly deploy a Rancher Server with a single node cluster a - Vagrant will require plugins to create VirtualBox VMs. Install them with the following commands: `vagrant plugin install vagrant-vboxmanage` - + `vagrant plugin install vagrant-vbguest` ## Getting Started @@ -32,7 +34,7 @@ The following steps quickly deploy a Rancher Server with a single node cluster a 4. To initiate the creation of the environment run, `vagrant up --provider=virtualbox`. -5. Once provisioning finishes, go to `https://172.22.101.101` in the browser. The default user/password is `admin/admin`. +5. Once provisioning finishes, go to `https://192.168.56.101` in the browser. The default user/password is `admin/admin`. **Result:** Rancher Server and your Kubernetes cluster is installed on VirtualBox. diff --git a/content/rancher/v2.5/en/security/_index.md b/content/rancher/v2.5/en/security/_index.md index c52daf4bde1..531c7d2ca33 100644 --- a/content/rancher/v2.5/en/security/_index.md +++ b/content/rancher/v2.5/en/security/_index.md @@ -33,6 +33,7 @@ On this page, we provide security-related documentation along with resources to - [The CIS Benchmark and self-assessment](#the-cis-benchmark-and-self-assessment) - [Third-party penetration test reports](#third-party-penetration-test-reports) - [Rancher CVEs and resolutions](#rancher-cves-and-resolutions) +- [Kubernetes Security Best Practices](#kubernetes-security-best-practices) ### Running a CIS Security Scan on a Kubernetes Cluster @@ -86,3 +87,7 @@ Results: ### Rancher CVEs and Resolutions Rancher is committed to informing the community of security issues in our products. For the list of CVEs (Common Vulnerabilities and Exposures) for issues we have resolved, refer to [this page.](./cve) + +### Kubernetes Security Best Practices + +For recommendations on securing your Kubernetes cluster, refer to the [Best Practices](./best-practices) guide. diff --git a/content/rancher/v2.5/en/security/best-practices/_index.md b/content/rancher/v2.5/en/security/best-practices/_index.md new file mode 100644 index 00000000000..1b207551e35 --- /dev/null +++ b/content/rancher/v2.5/en/security/best-practices/_index.md @@ -0,0 +1,8 @@ +--- +title: Kubernetes Security Best Practices +weight: 5 +--- + +# Restricting cloud metadata API access + +Cloud providers such as AWS, Azure, or GCP often expose metadata services locally to instances. By default, this endpoint is accessible by pods running on a cloud instance, including pods in hosted Kubernetes providers such as EKS, AKS or GKE, and can contain cloud credentials for that node, provisioning data such as kubelet credentials, or other sensitive data. To mitigate this risk when running on a cloud platform, follow the [Kubernetes security recommendations](https://kubernetes.io/docs/tasks/administer-cluster/securing-a-cluster/#restricting-cloud-metadata-api-access): limit permissions given to instance credentials, use network policies to restrict pod access to the metadata API, and avoid using provisioning data to deliver secrets. diff --git a/content/rancher/v2.5/en/security/rancher-2.5/1.5-benchmark-2.5/_index.md b/content/rancher/v2.5/en/security/rancher-2.5/1.5-benchmark-2.5/_index.md index 02b34e42fdb..4fed8f40550 100644 --- a/content/rancher/v2.5/en/security/rancher-2.5/1.5-benchmark-2.5/_index.md +++ b/content/rancher/v2.5/en/security/rancher-2.5/1.5-benchmark-2.5/_index.md @@ -1281,7 +1281,7 @@ on the master node and ensure the correct value for the `--bind-address` paramet **Expected result**: ``` -'--bind-address' is present OR '--bind-address' is not present +'--bind-address' argument is set to 127.0.0.1 ``` ### 1.4 Scheduler @@ -1327,7 +1327,7 @@ on the master node and ensure the correct value for the `--bind-address` paramet **Expected result**: ``` -'--bind-address' is present OR '--bind-address' is not present +'--bind-address' argument is set to 127.0.0.1 ``` ## 2 Etcd Node Configuration diff --git a/content/rancher/v2.5/en/security/rancher-2.5/1.5-hardening-2.5/_index.md b/content/rancher/v2.5/en/security/rancher-2.5/1.5-hardening-2.5/_index.md index 26907ab28b6..491aec9c080 100644 --- a/content/rancher/v2.5/en/security/rancher-2.5/1.5-hardening-2.5/_index.md +++ b/content/rancher/v2.5/en/security/rancher-2.5/1.5-hardening-2.5/_index.md @@ -667,6 +667,7 @@ rancher_kubernetes_engine_config: service_node_port_range: 30000-32767 kube_controller: extra_args: + bind-address: 127.0.0.1 address: 127.0.0.1 feature-gates: RotateKubeletServerCertificate=true profiling: 'false' @@ -685,6 +686,7 @@ rancher_kubernetes_engine_config: generate_serving_certificate: true scheduler: extra_args: + bind-address: 127.0.0.1 address: 127.0.0.1 profiling: 'false' ssh_agent_auth: false diff --git a/content/rancher/v2.5/en/security/rancher-2.5/1.6-benchmark-2.5/_index.md b/content/rancher/v2.5/en/security/rancher-2.5/1.6-benchmark-2.5/_index.md index 57b65e5b004..e0dc1e45c5d 100644 --- a/content/rancher/v2.5/en/security/rancher-2.5/1.6-benchmark-2.5/_index.md +++ b/content/rancher/v2.5/en/security/rancher-2.5/1.6-benchmark-2.5/_index.md @@ -1803,13 +1803,13 @@ on the master node and ensure the correct value for the --bind-address parameter **Expected Result**: ```console -'--bind-address' is not present OR '--bind-address' is not present +'--bind-address' argument is set to 127.0.0.1 ``` **Returned Value**: ```console -root 4788 4773 4 16:16 ? 00:00:09 kube-controller-manager --configure-cloud-routes=false --cloud-provider= --service-cluster-ip-range=10.43.0.0/16 --v=2 --pod-eviction-timeout=5m0s --leader-elect=true --cluster-cidr=10.42.0.0/16 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --profiling=false --node-monitor-grace-period=40s --service-account-private-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --address=0.0.0.0 --allow-untagged-cloud=true --root-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allocate-node-cidrs=true --enable-hostpath-provisioner=false --terminated-pod-gc-threshold=1000 --feature-gates=RotateKubeletServerCertificate=true --use-service-account-credentials=true +root 4788 4773 4 16:16 ? 00:00:09 kube-controller-manager --configure-cloud-routes=false --cloud-provider= --service-cluster-ip-range=10.43.0.0/16 --v=2 --bind-address=127.0.0.1 --pod-eviction-timeout=5m0s --leader-elect=true --cluster-cidr=10.42.0.0/16 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --profiling=false --node-monitor-grace-period=40s --service-account-private-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --address=127.0.0.1 --allow-untagged-cloud=true --root-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allocate-node-cidrs=true --enable-hostpath-provisioner=false --terminated-pod-gc-threshold=1000 --feature-gates=RotateKubeletServerCertificate=true --use-service-account-credentials=true ``` ## 1.4 Scheduler @@ -1859,13 +1859,13 @@ on the master node and ensure the correct value for the --bind-address parameter **Expected Result**: ```console -'--bind-address' is not present OR '--bind-address' is not present +'--bind-address' argument is set to 127.0.0.1 ``` **Returned Value**: ```console -root 4947 4930 1 16:16 ? 00:00:02 kube-scheduler --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-scheduler.yaml --leader-elect=true --profiling=false --v=2 --address=0.0.0.0 +root 4947 4930 1 16:16 ? 00:00:02 kube-scheduler --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-scheduler.yaml --leader-elect=true --profiling=false --v=2 --address=127.0.0.1 --bind-address=127.0.0.1 ``` ## 2 Etcd Node Configuration Files diff --git a/content/rancher/v2.5/en/security/rancher-2.5/1.6-hardening-2.5/_index.md b/content/rancher/v2.5/en/security/rancher-2.5/1.6-hardening-2.5/_index.md index 0b6497f2e1f..d628bfd8c5a 100644 --- a/content/rancher/v2.5/en/security/rancher-2.5/1.6-hardening-2.5/_index.md +++ b/content/rancher/v2.5/en/security/rancher-2.5/1.6-hardening-2.5/_index.md @@ -511,6 +511,8 @@ rancher_kubernetes_engine_config: kube_controller: extra_args: feature-gates: RotateKubeletServerCertificate=true + bind-address: 127.0.0.1 + address: 127.0.0.1 kubelet: extra_args: feature-gates: RotateKubeletServerCertificate=true @@ -519,6 +521,10 @@ rancher_kubernetes_engine_config: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 fail_swap_on: false generate_serving_certificate: true + scheduler: + extra_args: + bind-address: 127.0.0.1 + address: 127.0.0.1 ssh_agent_auth: false upgrade_strategy: max_unavailable_controlplane: '1' diff --git a/content/rancher/v2.5/en/troubleshooting/dns/_index.md b/content/rancher/v2.5/en/troubleshooting/dns/_index.md index 99274794206..3822399c32f 100644 --- a/content/rancher/v2.5/en/troubleshooting/dns/_index.md +++ b/content/rancher/v2.5/en/troubleshooting/dns/_index.md @@ -160,7 +160,7 @@ kubectl run -i --restart=Never --rm test-${RANDOM} --image=ubuntu --overrides='{ Enabling query logging can be done by enabling the [log plugin](https://coredns.io/plugins/log/) in the Corefile configuration in the configmap `coredns`. You can do so by using `kubectl -n kube-system edit configmap coredns` or use the command below to replace the configuration in place: ``` -kubectl get configmap -n kube-system coredns -o json | kubectl get configmap -n kube-system coredns -o json | sed -e 's_loadbalance_log\\n loadbalance_g' | kubectl apply -f - +kubectl get configmap -n kube-system coredns -o json | sed -e 's_loadbalance_log\\n loadbalance_g' | kubectl apply -f - ``` All queries will now be logged and can be checked using the command in [Check CoreDNS logging](#check-coredns-logging). diff --git a/content/rancher/v2.6/_index.md b/content/rancher/v2.6/_index.md index a1ea29d36e4..5060194d87d 100644 --- a/content/rancher/v2.6/_index.md +++ b/content/rancher/v2.6/_index.md @@ -1,5 +1,5 @@ --- -title: v2.6.x +title: Rancher 2.6 weight: 1 showBreadcrumb: false --- diff --git a/content/rancher/v2.6/en/_index.md b/content/rancher/v2.6/en/_index.md index d8bfd4a20c3..98bdedcac77 100644 --- a/content/rancher/v2.6/en/_index.md +++ b/content/rancher/v2.6/en/_index.md @@ -2,7 +2,7 @@ title: "Rancher 2.6" shortTitle: "Rancher 2.6 (Latest)" description: "Rancher adds significant value on top of Kubernetes: managing hundreds of clusters from one interface, centralizing RBAC, enabling monitoring and alerting. Read more." -metaTitle: "Rancher 2.x Docs: What is New?" +metaTitle: "Rancher 2.6 Docs: What is New?" metaDescription: "Rancher 2 adds significant value on top of Kubernetes: managing hundreds of clusters from one interface, centralizing RBAC, enabling monitoring and alerting. Read more." insertOneSix: false weight: 1 diff --git a/content/rancher/v2.6/en/admin-settings/authentication/ad/_index.md b/content/rancher/v2.6/en/admin-settings/authentication/ad/_index.md index a919075bdaa..251039f2c97 100644 --- a/content/rancher/v2.6/en/admin-settings/authentication/ad/_index.md +++ b/content/rancher/v2.6/en/admin-settings/authentication/ad/_index.md @@ -101,7 +101,7 @@ The table below details the parameters for the group schema configuration. | Search Attribute | Attribute used to construct search filters when adding groups to clusters or projects. See description of user schema `Search Attribute`. | | Search Filter | This filter gets applied to the list of groups that is searched when Rancher attempts to add groups to a site access list or tries to add groups to clusters or projects. For example, a group search filter could be (|(cn=group1)(cn=group2)). Note: If the search filter does not use [valid AD search syntax,](https://docs.microsoft.com/en-us/windows/win32/adsi/search-filter-syntax) the list of groups will be empty. | | Group DN Attribute | The name of the group attribute whose format matches the values in the user attribute describing a the user's memberships. See `User Member Attribute`. | -| Nested Group Membership | This settings defines whether Rancher should resolve nested group memberships. Use only if your organisation makes use of these nested memberships (ie. you have groups that contain other groups as members. We advise avoiding nested groups when possible). | +| Nested Group Membership | This settings defines whether Rancher should resolve nested group memberships. Use only if your organization makes use of these nested memberships (i.e., you have groups that contain other groups as members. We advise avoiding nested groups when possible to avoid potential performance issues when there is a large amount of nested memberships). | --- @@ -153,7 +153,7 @@ This command performs an LDAP search with the search base set to the domain root Since in this case the user's DN is `CN=John Doe,CN=Users,DC=acme,DC=com` [5], we should configure the **User Search Base** with the parent node DN `CN=Users,DC=acme,DC=com`. -Similarly, based on the DN of the group referenced in the **memberOf** attribute [4], the correct value for the **Group Search Base** would be the parent node of that value, ie. `OU=Groups,DC=acme,DC=com`. +Similarly, based on the DN of the group referenced in the **memberOf** attribute [4], the correct value for the **Group Search Base** would be the parent node of that value, i.e., `OU=Groups,DC=acme,DC=com`. ### Identify User Schema @@ -166,7 +166,7 @@ The output of the above `ldapsearch` query also allows to determine the correct > **Note:** > -> If the AD users in our organisation were to authenticate with their UPN (e.g. jdoe@acme.com) instead of the short logon name, then we would have to set the `Login Attribute` to **userPrincipalName** instead. +> If the AD users in our organization were to authenticate with their UPN (e.g. jdoe@acme.com) instead of the short logon name, then we would have to set the `Login Attribute` to **userPrincipalName** instead. We'll also set the `Search Attribute` parameter to **sAMAccountName|name**. That way users can be added to clusters/projects in the Rancher UI either by entering their username or full name. diff --git a/content/rancher/v2.6/en/admin-settings/authentication/keycloak-saml/_index.md b/content/rancher/v2.6/en/admin-settings/authentication/keycloak-saml/_index.md index bd94b700d23..ca2952111fb 100644 --- a/content/rancher/v2.6/en/admin-settings/authentication/keycloak-saml/_index.md +++ b/content/rancher/v2.6/en/admin-settings/authentication/keycloak-saml/_index.md @@ -23,36 +23,87 @@ If your organization uses Keycloak Identity Provider (IdP) for user authenticati >1: Optionally, you can enable either one or both of these settings. >2: Rancher SAML metadata won't be generated until a SAML provider is configured and saved. - + {{< img "/img/rancher/keycloak/keycloak-saml-client-configuration.png" "">}} - + - In the new SAML client, create Mappers to expose the users fields - Add all "Builtin Protocol Mappers" {{< img "/img/rancher/keycloak/keycloak-saml-client-builtin-mappers.png" "">}} - Create a new "Group list" mapper to map the member attribute to a user's groups - {{< img "/img/rancher/keycloak/keycloak-saml-client-group-mapper.png" "">}} -- Export a `metadata.xml` file from your Keycloak client: - From the `Installation` tab, choose the `SAML Metadata IDPSSODescriptor` format option and download your file. - - >**Note** - > Keycloak versions 6.0.0 and up no longer provide the IDP metadata under the `Installation` tab. - > You can still get the XML from the following url: - > - > `https://{KEYCLOAK-URL}/auth/realms/{REALM-NAME}/protocol/saml/descriptor` - > - > The XML obtained from this URL contains `EntitiesDescriptor` as the root element. Rancher expects the root element to be `EntityDescriptor` rather than `EntitiesDescriptor`. So before passing this XML to Rancher, follow these steps to adjust it: - > - > * Copy all the attributes from `EntitiesDescriptor` to the `EntityDescriptor` that are not present. - > * Remove the `` tag from the beginning. - > * Remove the `` from the end of the xml. - > - > You are left with something similar as the example below: - > - > ``` - > - > .... - > - > ``` + {{< img "/img/rancher/keycloak/keycloak-saml-client-group-mapper.png" "">}} + +## Getting the IDP Metadata + +{{% tabs %}} +{{% tab "Keycloak 5 and earlier" %}} +To get the IDP metadata, export a `metadata.xml` file from your Keycloak client. +From the **Installation** tab, choose the **SAML Metadata IDPSSODescriptor** format option and download your file. +{{% /tab %}} +{{% tab "Keycloak 6-13" %}} + +1. From the **Configure** section, click the **Realm Settings** tab. +1. Click the **General** tab. +1. From the **Endpoints** field, click **SAML 2.0 Identity Provider Metadata**. + +Verify the IDP metadata contains the following attributes: + +``` +xmlns:md="urn:oasis:names:tc:SAML:2.0:metadata" +xmlns:saml="urn:oasis:names:tc:SAML:2.0:assertion" +xmlns:ds="http://www.w3.org/2000/09/xmldsig#" +``` + +Some browsers, such as Firefox, may render/process the document such that the contents appear to have been modified, and some attributes appear to be missing. In this situation, use the raw response data that can be found using your browser. + +The following is an example process for Firefox, but will vary slightly for other browsers: + +1. Press **F12** to access the developer console. +1. Click the **Network** tab. +1. From the table, click the row containing `descriptor`. +1. From the details pane, click the **Response** tab. +1. Copy the raw response data. + +The XML obtained contains `EntitiesDescriptor` as the root element. Rancher expects the root element to be `EntityDescriptor` rather than `EntitiesDescriptor`. So before passing this XML to Rancher, follow these steps to adjust it: + +1. Copy all the attributes from `EntitiesDescriptor` to the `EntityDescriptor` that are not present. +1. Remove the `` tag from the beginning. +1. Remove the `` from the end of the xml. + +You are left with something similar as the example below: + +``` + +.... + +``` + +{{% /tab %}} +{{% tab "Keycloak 14+" %}} + +1. From the **Configure** section, click the **Realm Settings** tab. +1. Click the **General** tab. +1. From the **Endpoints** field, click **SAML 2.0 Identity Provider Metadata**. + +Verify the IDP metadata contains the following attributes: + +``` +xmlns:md="urn:oasis:names:tc:SAML:2.0:metadata" +xmlns:saml="urn:oasis:names:tc:SAML:2.0:assertion" +xmlns:ds="http://www.w3.org/2000/09/xmldsig#" +``` + +Some browsers, such as Firefox, may render/process the document such that the contents appear to have been modified, and some attributes appear to be missing. In this situation, use the raw response data that can be found using your browser. + +The following is an example process for Firefox, but will vary slightly for other browsers: + +1. Press **F12** to access the developer console. +1. Click the **Network** tab. +1. From the table, click the row containing `descriptor`. +1. From the details pane, click the **Response** tab. +1. Copy the raw response data. + +{{% /tab %}} +{{% /tabs %}} ## Configuring Keycloak in Rancher diff --git a/content/rancher/v2.6/en/admin-settings/authentication/ping-federate/_index.md b/content/rancher/v2.6/en/admin-settings/authentication/ping-federate/_index.md index 5040d63561b..484b0a1610c 100644 --- a/content/rancher/v2.6/en/admin-settings/authentication/ping-federate/_index.md +++ b/content/rancher/v2.6/en/admin-settings/authentication/ping-federate/_index.md @@ -27,6 +27,8 @@ Note that these URLs will not return valid data until the authentication configu 1. **Groups Field**: Make entries for managing group memberships (example: `memberOf`). + 1. **Entity ID Field** (optional): The published, protocol-dependent, unique identifier of your partner. This ID defines your organization as the entity operating the server for SAML 2.0 transactions. This ID may have been obtained out-of-band or via a SAML metadata file. + 1. **Rancher API Host**: Enter the URL for your Rancher Server. 1. **Private Key** and **Certificate**: This is a key-certificate pair to create a secure shell between Rancher and your IdP. diff --git a/content/rancher/v2.6/en/admin-settings/branding/_index.md b/content/rancher/v2.6/en/admin-settings/branding/_index.md index 1fbe6d9f78b..4e5cff17e20 100644 --- a/content/rancher/v2.6/en/admin-settings/branding/_index.md +++ b/content/rancher/v2.6/en/admin-settings/branding/_index.md @@ -40,7 +40,21 @@ You can override the primary color used throughout the UI with a custom color of ### Fixed Banners +{{% tabs %}} +{{% tab "Rancher before v2.6.4" %}} Display a custom fixed banner in the header, footer, or both. +{{% /tab %}} +{{% tab "Rancher v2.6.4+" %}} +Display a custom fixed banner in the header, footer, or both. + +As of Rancher v2.6.4, configuration of fixed banners has moved from the **Branding** tab to the **Banners** tab. + +To configure banner settings, + +1. Click **☰ > Global settings**. +2. Click **Banners**. +{{% /tab %}} +{{% /tabs %}} # Custom Navigation Links diff --git a/content/rancher/v2.6/en/admin-settings/config-private-registry/_index.md b/content/rancher/v2.6/en/admin-settings/config-private-registry/_index.md index 16690af8bd3..cd36201d114 100644 --- a/content/rancher/v2.6/en/admin-settings/config-private-registry/_index.md +++ b/content/rancher/v2.6/en/admin-settings/config-private-registry/_index.md @@ -3,7 +3,7 @@ title: Configuring a Global Default Private Registry weight: 40 --- -You might want to use a private Docker registry to share your custom base images within your organization. With a private registry, you can keep a private, consistent, and centralized source of truth for the container images that are used in your clusters. +You might want to use a private container registry to share your custom base images within your organization. With a private registry, you can keep a private, consistent, and centralized source of truth for the container images that are used in your clusters. There are two main ways to set up private registries in Rancher: by setting up the global default registry through the **Settings** tab in the global view, and by setting up a private registry in the advanced options in the cluster-level settings. The global default registry is intended to be used for air-gapped setups, for registries that do not require credentials. The cluster-level private registry is intended to be used in all setups in which the private registry requires credentials. @@ -35,4 +35,4 @@ You can follow these steps to configure a private registry when you create a clu **Result:** The new cluster will be able to pull images from the private registry. -The private registry cannot be configured after the cluster is created. \ No newline at end of file +The private registry cannot be configured after the cluster is created. diff --git a/content/rancher/v2.6/en/admin-settings/pod-security-policies/_index.md b/content/rancher/v2.6/en/admin-settings/pod-security-policies/_index.md index 2e0d6d70fc3..9244102623d 100644 --- a/content/rancher/v2.6/en/admin-settings/pod-security-policies/_index.md +++ b/content/rancher/v2.6/en/admin-settings/pod-security-policies/_index.md @@ -9,6 +9,7 @@ If a pod does not meet the conditions specified in the PSP, Kubernetes will not - [How PSPs Work](#how-psps-work) - [Default PSPs](#default-psps) + - [Restricted-NoRoot](#restricted-noroot) - [Restricted](#restricted) - [Unrestricted](#unrestricted) - [Creating PSPs](#creating-psps) @@ -28,18 +29,22 @@ PSPs work through inheritance: Any workloads that are already running in a cluster or project before a PSP is assigned will not be checked if it complies with the PSP. Workloads would need to be cloned or upgraded to see if they pass the PSP. -Read more about Pod Security Policies in the [Kubernetes Documentation](https://kubernetes.io/docs/concepts/policy/pod-security-policy/). +Read more about Pod Security Policies in the [Kubernetes documentation](https://kubernetes.io/docs/concepts/policy/pod-security-policy/). # Default PSPs -Rancher ships with two default Pod Security Policies (PSPs): the `restricted` and `unrestricted` policies. +Rancher ships with three default Pod Security Policies (PSPs): the `restricted-noroot`, `restricted` and `unrestricted` policies. -### Restricted +### Restricted-NoRoot This policy is based on the Kubernetes [example restricted policy](https://raw.githubusercontent.com/kubernetes/website/master/content/en/examples/policy/restricted-psp.yaml). It significantly restricts what types of pods can be deployed to a cluster or project. This policy: - Prevents pods from running as a privileged user and prevents escalation of privileges. -- Validates that server-required security mechanisms are in place (such as restricting what volumes can be mounted to only the core volume types and preventing root supplemental groups from being added. +- Validates that server-required security mechanisms are in place, such as restricting what volumes can be mounted to only the core volume types and preventing root supplemental groups from being added. + +### Restricted + +This policy is a relaxed version of the `restricted-noroot` policy, with almost all the restrictions in place, except for the fact that it allows running containers as a privileged user. ### Unrestricted @@ -51,9 +56,9 @@ Using Rancher, you can create a Pod Security Policy using our GUI rather than cr ### Requirements -Rancher can only assign PSPs for clusters that are [launched using RKE.]({{< baseurl >}}/rancher/v2.6/en/cluster-provisioning/rke-clusters/) +Rancher can only assign PSPs for clusters that are [launched using RKE]({{< baseurl >}}/rancher/v2.6/en/cluster-provisioning/rke-clusters/). -You must enable PSPs at the cluster level before you can assign them to a project. This can be configured by [editing the cluster.]({{}}/rancher/v2.6/en/cluster-admin/editing-clusters/) +You must enable PSPs at the cluster level before you can assign them to a project. This can be configured by [editing the cluster]({{}}/rancher/v2.6/en/cluster-admin/editing-clusters/). It is a best practice to set PSP at the cluster level. @@ -63,19 +68,11 @@ We recommend adding PSPs during cluster and project creation instead of adding i 1. In the upper left corner, click **☰ > Cluster Management**. 1. In the left navigation bar, click **Pod Security Policies**. -1. Click **Add policy**. +1. Click **Add Policy**. 1. Name the policy. 1. Complete each section of the form. Refer to the [Kubernetes documentation](https://kubernetes.io/docs/concepts/policy/pod-security-policy/) for more information on what each policy does. 1. Click **Create**. # Configuration -The Kubernetes documentation on PSPs is [here.](https://kubernetes.io/docs/concepts/policy/pod-security-policy/) - - - - - -[1]: https://kubernetes.io/docs/concepts/policy/pod-security-policy/#volumes-and-file-systems -[2]: https://kubernetes.io/docs/concepts/policy/pod-security-policy/#host-namespaces -[3]: https://kubernetes.io/docs/concepts/policy/pod-security-policy/#users-and-groups +The Kubernetes documentation on PSPs is [here](https://kubernetes.io/docs/concepts/policy/pod-security-policy/). diff --git a/content/rancher/v2.6/en/admin-settings/rbac/cluster-project-roles/_index.md b/content/rancher/v2.6/en/admin-settings/rbac/cluster-project-roles/_index.md index c966d787383..c2767b8a66f 100644 --- a/content/rancher/v2.6/en/admin-settings/rbac/cluster-project-roles/_index.md +++ b/content/rancher/v2.6/en/admin-settings/rbac/cluster-project-roles/_index.md @@ -5,7 +5,7 @@ weight: 1127 Cluster and project roles define user authorization inside a cluster or project. -To manage these roles, +To manage these roles, 1. Click **☰ > Users & Authentication**. 1. In the left navigation bar, click **Roles** and go to the **Cluster** or **Project/Namespaces** tab. @@ -60,8 +60,8 @@ The following table lists the permissions available for the `Manage Nodes` role | SSH Access | ✓ | ✓ | | Delete Nodes | ✓ | ✓ | | Scale Clusters Up and Down | ✓ | * | -***In RKE2, you must have permission to edit a cluster to be able to scale clusters up and down.** -
+***In RKE2, you must have permission to edit a cluster to be able to scale clusters up and down.** +
For details on how each cluster role can access Kubernetes resources, you can look them up in the Rancher UI: @@ -81,12 +81,24 @@ To assign a custom role to a new cluster member, you can use the Rancher UI. To To assign the role to a new cluster member, +{{% tabs %}} +{{% tab "Rancher before v2.6.4" %}} 1. Click **☰ > Cluster Management**. 1. Go to the cluster where you want to assign a role to a member and click **Explore**. 1. Click **RBAC > Cluster Members**. 1. Click **Add**. 1. In the **Cluster Permissions** section, choose the custom cluster role that should be assigned to the member. 1. Click **Create**. +{{% /tab %}} +{{% tab "Rancher v2.6.4+" %}} +1. Click **☰ > Cluster Management**. +1. Go to the cluster where you want to assign a role to a member and click **Explore**. +1. Click **Cluster > Cluster Members**. +1. Click **Add**. +1. In the **Cluster Permissions** section, choose the custom cluster role that should be assigned to the member. +1. Click **Create**. +{{% /tab %}} +{{% /tabs %}} **Result:** The member has the assigned role. @@ -111,6 +123,10 @@ _Project roles_ are roles that can be used to grant users access to a project. T These users can manage project-scoped resources like namespaces and workloads, but cannot manage other project members. + >**Note:** + > + >By default, the Rancher role of `project-member` inherits from the `Kubernetes-edit` role, and the `project-owner` role inherits from the `Kubernetes-admin` role. As such, both `project-member` and `project-owner` roles will allow for namespace management, including the ability to create and delete namespaces. + - **Read Only:** These users can view everything in the project but cannot create, update, or delete anything. @@ -119,7 +135,6 @@ _Project roles_ are roles that can be used to grant users access to a project. T > >Users assigned the `Owner` or `Member` role for a project automatically inherit the `namespace creation` role. However, this role is a [Kubernetes ClusterRole](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#role-and-clusterrole), meaning its scope extends to all projects in the cluster. Therefore, users explicitly assigned the `owner` or `member` role for a project can create namespaces in other projects they're assigned to, even with only the `Read Only` role assigned. - #### Custom Project Roles Rancher lets you assign _custom project roles_ to a standard user instead of the typical `Owner`, `Member`, or `Read Only` roles. These roles can be either a built-in custom project role or one defined by a Rancher administrator. They are convenient for defining narrow or specialized access for a standard user within a project. See the table below for a list of built-in custom project roles. diff --git a/content/rancher/v2.6/en/admin-settings/rke-templates/enforcement/_index.md b/content/rancher/v2.6/en/admin-settings/rke-templates/enforcement/_index.md index 9125f7ff342..cd51f196341 100644 --- a/content/rancher/v2.6/en/admin-settings/rke-templates/enforcement/_index.md +++ b/content/rancher/v2.6/en/admin-settings/rke-templates/enforcement/_index.md @@ -25,6 +25,8 @@ To require new clusters to use an RKE template, administrators can turn on RKE t 1. Go to the `cluster-template-enforcement` setting. Click **⋮ > Edit Setting**. 1. Set the value to **True** and click **Save**. + >**Important:** When the admin sets the `cluster-template-enforcement` to True, they also need to share the `clusterTemplates` with users so that users can select one of these templates to create the cluster. + **Result:** All clusters provisioned by Rancher must use a template, unless the creator is an administrator. # Disabling RKE Template Enforcement diff --git a/content/rancher/v2.6/en/api/_index.md b/content/rancher/v2.6/en/api/_index.md index 60cf9320914..d1cc9cc4454 100644 --- a/content/rancher/v2.6/en/api/_index.md +++ b/content/rancher/v2.6/en/api/_index.md @@ -5,7 +5,22 @@ weight: 24 ## How to use the API -The API has its own user interface accessible from a web browser. This is an easy way to see resources, perform actions, and see the equivalent cURL or HTTP request & response. To access it, click on your user avatar in the upper right corner. Under **API & Keys**, you can find the URL endpoint as well as create [API keys]({{}}/rancher/v2.6/en/user-settings/api-keys/). +The API has its own user interface accessible from a web browser. This is an easy way to see resources, perform actions, and see the equivalent cURL or HTTP request & response. To access it: + +{{% tabs %}} +{{% tab "Rancher v2.6.4+" %}} + +1. Click on your user avatar in the upper right corner. +1. Click **Account & API Keys**. +1. Under the **API Keys** section, find the **API Endpoint** field and click the link. The link will look something like `https:///v3`, where `` is the fully qualified domain name of your Rancher deployment. + +{{% /tab %}} +{{% tab "Rancher before v2.6.4" %}} + +Go to the URL endpoint at `https:///v3`, where `` is the fully qualified domain name of your Rancher deployment. + +{{% /tab %}} +{{% /tabs %}} ## Authentication @@ -50,3 +65,17 @@ Most collections can be sorted on the server-side by common fields using HTTP qu ## Pagination API responses are paginated with a limit of 100 resources per page by default. This can be changed with the `limit` query parameter, up to a maximum of 1000, e.g. `/v3/pods?limit=1000`. The `pagination` map in collection responses tells you whether or not you have the full result set and has a link to the next page if you do not. + +## Capturing Rancher API Calls + +You can use browser developer tools to capture how the Rancher API is called. For example, you could follow these steps to use the Chrome developer tools to get the API call for provisioning an RKE cluster: + +1. In the Rancher UI, go to **Cluster Management** and click **Create.** +1. Click one of the cluster types. This example uses Digital Ocean. +1. Fill out the form with a cluster name and node template, but don't click **Create**. +1. You will need to open the developer tools before the cluster creation to see the API call being recorded. To open the tools, right-click on the Rancher UI and click **Inspect.** +1. In the developer tools, click the **Network** tab. +1. On the **Network** tab, make sure **Fetch/XHR** is selected. +1. In the Rancher UI, click **Create**. In the developer tools, you should see a new network request with the name `cluster?_replace=true`. +1. Right-click `cluster?_replace=true` and click **Copy > Copy as cURL.** +1. Paste the result into any text editor. You will be able to see the POST request, including the URL it was sent to, all of the headers, and the full body of the request. This command can be used to create a cluster from the command line. Note: The request should be stored in a safe place because it contains credentials. diff --git a/content/rancher/v2.6/en/backups/_index.md b/content/rancher/v2.6/en/backups/_index.md index b090f2972b2..d9e4dd96065 100644 --- a/content/rancher/v2.6/en/backups/_index.md +++ b/content/rancher/v2.6/en/backups/_index.md @@ -43,10 +43,14 @@ When a Restore custom resource is created, the operator accesses the backup .tar The Backup and Restore custom resources can be created in the Rancher UI, or by using `kubectl apply`. +>**Note:** Refer [here]({{}}/rancher/v2.6/en/backups/migrating-rancher/#2-restore-from-backup-using-a-restore-custom-resource) for help on restoring an existing backup file into a v1.22 cluster in Rancher v2.6.3. + # Installing the rancher-backup Operator The `rancher-backup` operator can be installed from the Rancher UI, or with the Helm CLI. In both cases, the `rancher-backup` Helm chart is installed on the Kubernetes cluster running the Rancher server. It is a cluster-admin only feature and available only for the **local** cluster. (*If you do not see `rancher-backup` in the Rancher UI, you may have selected the wrong cluster.*) +>**NOTE:** There is a known issue in Fleet that occurs after performing a restoration using the backup-restore-operator: Secrets used for clientSecretName and helmSecretName are not included in Fleet gitrepos. Refer [here]({{}}rancher/v2.6/en/deploy-across-clusters/fleet/#troubleshooting) for a workaround. + ### Installing rancher-backup with the Rancher UI 1. In the upper left corner, click **☰ > Cluster Management**. diff --git a/content/rancher/v2.6/en/backups/back-up-rancher/_index.md b/content/rancher/v2.6/en/backups/back-up-rancher/_index.md index acfe23fc3fb..629a7ad2c64 100644 --- a/content/rancher/v2.6/en/backups/back-up-rancher/_index.md +++ b/content/rancher/v2.6/en/backups/back-up-rancher/_index.md @@ -5,12 +5,18 @@ weight: 1 In this section, you'll learn how to back up Rancher running on any Kubernetes cluster. To backup Rancher installed with Docker, refer the instructions for [single node backups]({{}}/rancher/v2.6/en/backups/docker-installs/docker-backups) -The backup-restore operator needs to be installed in the local cluster, and only backs up the Rancher app. The backup and restore operations are performed only in the local Kubernetes cluster. +The backup-restore operator needs to be installed in the local cluster, and only backs up the Rancher app. The backup and restore operations are performed only in the local Kubernetes cluster. + +Note that the rancher-backup operator version 2.x.x is for Rancher v2.6.x. + +> When restoring a backup into a new Rancher setup, the version of the new setup should be the same as the one where the backup is made. The Kubernetes version should also be considered when restoring a backup, since the supported apiVersion in the cluster and in the backup file could be different. ### Prerequisites The Rancher version must be v2.5.0 and up. +Refer [here]({{}}/rancher/v2.6/en/backups/migrating-rancher/#2-restore-from-backup-using-a-restore-custom-resource) for help on restoring an existing backup file into a v1.22 cluster in Rancher v2.6.3. + ### 1. Install the Rancher Backups operator The backup storage location is an operator-level setting, so it needs to be configured when the Rancher Backups application is installed or upgraded. @@ -25,6 +31,8 @@ Backups are created as .tar.gz files. These files can be pushed to S3 or Minio, 1. Configure the default storage location. For help, refer to the [storage configuration section.](../configuration/storage-config) 1. Click **Install**. +>**NOTE:** There is a known issue in Fleet that occurs after performing a restoration using the backup-restore-operator: Secrets used for clientSecretName and helmSecretName are not included in Fleet gitrepos. Refer [here]({{}}rancher/v2.6/en/deploy-across-clusters/fleet/#troubleshooting) for a workaround. + ### 2. Perform a Backup To perform a backup, a custom resource of type Backup must be created. diff --git a/content/rancher/v2.6/en/backups/examples/_index.md b/content/rancher/v2.6/en/backups/examples/_index.md index 87c607b7780..b1282f9c3ca 100644 --- a/content/rancher/v2.6/en/backups/examples/_index.md +++ b/content/rancher/v2.6/en/backups/examples/_index.md @@ -272,6 +272,13 @@ data: # Example EncryptionConfiguration +The snippet below demonstrates two different types of secrets and their relevance with respect to Backup and Restore of custom resources. + +The first example is that of a secret that is used to encrypt the backup files. The backup operator, in this case, will not be able to read the secrets encryption file. It only uses the contents of the secret. + +The second example is that of a Kubernetes secrets encryption config file that is used to encrypt secrets when stored in etcd. **When backing up the etcd datastore, be sure to also back up the EncryptionConfiguration.** Failure to do so will result in an inability to use the restored data if secrets encryption was in use at the time the data was backed up. + + ```yaml apiVersion: apiserver.config.k8s.io/v1 kind: EncryptionConfiguration diff --git a/content/rancher/v2.6/en/backups/migrating-rancher/_index.md b/content/rancher/v2.6/en/backups/migrating-rancher/_index.md index 0154b6b1bd7..5b5e4f3acdd 100644 --- a/content/rancher/v2.6/en/backups/migrating-rancher/_index.md +++ b/content/rancher/v2.6/en/backups/migrating-rancher/_index.md @@ -9,7 +9,7 @@ If you are migrating Rancher to a new Kubernetes cluster, you don't need to inst These instructions assume you have [created a backup](../back-up-rancher) and you have already installed a new Kubernetes cluster where Rancher will be deployed. -It is required to use the same hostname that was set as the server URL in the first cluster. +>**Warning:** It is required to use the same hostname that was set as the server URL in the first cluster. If not done, downstream clusters will show as unavailable in the cluster management page of the UI, and you won't be able to click inside the cluster or on the cluster's Explore button. Rancher version must be v2.5.0 and up @@ -19,17 +19,31 @@ Rancher can be installed on any Kubernetes cluster, including hosted Kubernetes - [K3s Kubernetes installation docs]({{}}/k3s/latest/en/installation/) ### 1. Install the rancher-backup Helm chart -Install version 2.x.x of the rancher-backup chart. +Install version 2.x.x of the rancher-backup chart. The following assumes a connected environment with access to DockerHub: ``` helm repo add rancher-charts https://charts.rancher.io helm repo update helm install rancher-backup-crd rancher-charts/rancher-backup-crd -n cattle-resources-system --create-namespace --version $CHART_VERSION helm install rancher-backup rancher-charts/rancher-backup -n cattle-resources-system --version $CHART_VERSION ``` +
+For an **air-gapped environment**, use the option below to pull the `backup-restore-operator` image from your private registry when installing the rancher-backup-crd helm chart. +``` +--set image.repository $REGISTRY/rancher/backup-restore-operator +``` ### 2. Restore from backup using a Restore custom resource -If you are using an S3 store as the backup source, and need to use your S3 credentials for restore, create a secret in this cluster using your S3 credentials. The Secret data must have two keys, `accessKey` and `secretKey` containing the s3 credentials like this: +>**Important:** Kubernetes v1.22, available as an experimental feature of v2.6.3, does not support restoring from backup files containing CRDs with the apiVersion `apiextensions.k8s.io/v1beta1`. In v1.22, the default `resourceSet` in the rancher-backup app is updated to collect only CRDs that use `apiextensions.k8s.io/v1`. There are currently two ways to work around this issue: +> +1. Update the default `resourceSet` to collect the CRDs with the apiVersion v1. +1. Update the default `resourceSet` and the client to use the new APIs internally, with `apiextensions.k8s.io/v1` as the replacement. +> +> - Note that when making or restoring backups for v1.22, the Rancher version and the local cluster's Kubernetes version should be the same. The Kubernetes version should be considered when restoring a backup since the supported apiVersion in the cluster and in the backup file could be different. + +If you are using an S3 store as the backup source and need to use your S3 credentials for restore, create a secret in this cluster using your S3 credentials. The Secret data must have two keys - `accessKey` and `secretKey` - that contain the S3 credentials. + +**Warning:** The values `accessKey` and `secretKey` in the example below must be base64-encoded first when creating the object directly. If not encoded first, the pasted values will cause errors when you are attempting to backup or restore. ```yaml apiVersion: v1 @@ -42,7 +56,7 @@ stringData: secretKey: ``` -This secret can be created in any namespace, with the above example it will get created in the default namespace +This secret can be created in any namespace; with the above example, it will get created in the default namespace. In the Restore custom resource, `prune` must be set to false. @@ -68,21 +82,34 @@ spec: endpoint: s3.us-west-2.amazonaws.com ``` -> **Important:** The field `encryptionConfigSecretName` must be set only if your backup was created with encryption enabled. Provide the name of the Secret containing the encryption config file. If you only have the encryption config file, but don't have a secret created with it in this cluster, use the following steps to create the secret: -1. The encryption configuration file must be named `encryption-provider-config.yaml`, and the `--from-file` flag must be used to create this secret. So save your `EncryptionConfiguration` in a file called `encryption-provider-config.yaml` and run this command: +>**Important:** The field `encryptionConfigSecretName` must be set only if your backup was created with encryption enabled. Provide the name of the Secret containing the encryption config file. If you only have the encryption config file, but don't have a secret created with it in this cluster, use the following steps to create the secret: +1. The encryption configuration file must be named `encryption-provider-config.yaml`, and the `--from-file` flag must be used to create this secret. So save your `EncryptionConfiguration` in a file called `encryption-provider-config.yaml` and run this command: ``` kubectl create secret generic encryptionconfig \ --from-file=./encryption-provider-config.yaml \ -n cattle-resources-system ``` -Then apply the resource: +1. Apply the manifest, and watch for the Restore resources status: + Apply the resource: ``` -kubectl apply -f migrationResource.yaml +kubectl apply -f migrationResource.yaml ``` + Watch the Restore status: +``` +kubectl get restore +``` + + Watch the restoration logs: +``` +kubectl logs -n cattle-resources-system --tail 100 -f rancher-backup-xxx-xxx +``` + +Once the Restore resource has the status `Completed`, you can continue the Rancher installation. + ### 3. Install cert-manager Follow the steps to [install cert-manager]({{}}/rancher/v2.6/en/installation/install-rancher-on-k8s/#5-install-cert-manager) in the documentation about installing cert-manager on Kubernetes. diff --git a/content/rancher/v2.6/en/backups/restoring-rancher/_index.md b/content/rancher/v2.6/en/backups/restoring-rancher/_index.md index 0d9925329a1..2ec7691aa79 100644 --- a/content/rancher/v2.6/en/backups/restoring-rancher/_index.md +++ b/content/rancher/v2.6/en/backups/restoring-rancher/_index.md @@ -3,15 +3,33 @@ title: Restoring Rancher weight: 2 --- -A restore is performed by creating a Restore custom resource. +This page outlines how to perform a restore with Rancher. > **Important** > > * Follow the instructions from this page for restoring rancher on the same cluster where it was backed up from. In order to migrate rancher to a new cluster, follow the steps to [migrate rancher.]({{}}/rancher/v2.6/en/backups/migrating-rancher) > * While restoring rancher on the same setup, the operator will scale down the rancher deployment when restore starts, and it will scale back up the deployment once restore completes. So Rancher will be unavailable during the restore. +> * If you need to restore Rancher to a previous version after an upgrade, see the [rollback documentation.]({{}}/rancher/v2.6/en/installation/install-rancher-on-k8s/rollbacks/) + +### Additional Steps for Rollbacks with Rancher v2.6.4+ + +In Rancher v2.6.4, the cluster-api module has been upgraded from v0.4.4 to v1.0.2 in which the apiVersion of CAPI CRDs are upgraded from `cluster.x-k8s.io/v1alpha4` to `cluster.x-k8s.io/v1beta1`. This has the effect of causing rollbacks from Rancher v2.6.4 to any previous version of Rancher v2.6.x to fail because the previous version the CRDs needed to roll back are no longer available in v1beta1. + +To avoid this, the Rancher resource cleanup scripts should be run **before** the restore or rollback is attempted. Specifically, two scripts have been created to assist you: one to clean up the cluster (`cleanup.sh`), and one to check for any Rancher-related resources in the cluster (`verify.sh`). Details on the cleanup script can be found in the [rancherlabs/support-tools repo](https://github.com/rancherlabs/support-tools/tree/master/rancher-cleanup). + +> **Caution** Rancher will be down as the `cleanup` script runs as it deletes the resources created by rancher. + +The additional preparations: + +1. Follow these [instructions](https://github.com/rancherlabs/support-tools/blob/master/rancher-cleanup/README.md) to run the scripts. +1. Follow these [instructions](https://rancher.com/docs/rancher/v2.6/en/backups/migrating-rancher/) to install the rancher-backup Helm chart on the existing cluster and restore the previous state. + 1. Omit Step 3. + 1. When Step 4 is reached, install the required Rancher v2.6.x version on the local cluster you intend to roll back to. ### Create the Restore Custom Resource +A restore is performed by creating a Restore custom resource. + 1. In the upper left corner, click **☰ > Cluster Management**. 1. On the **Clusters** page, go to the `local` cluster and click **Explore**. The `local` cluster runs the Rancher server. 1. In the left navigation bar, click **Rancher Backups > Restores**. @@ -64,6 +82,6 @@ In some cases, after restoring the backup, Rancher logs will show errors similar ``` 2021/10/05 21:30:45 [ERROR] error syncing 'c-89d82/m-4067aa68dd78': handler rke-worker-upgrader: clusters.management.cattle.io "c-89d82" not found, requeuing ``` -This happens because one of the resources that was just restored has finalizers but the related resources have been deleted so the handler cannot find it. +This happens because one of the resources that was just restored has finalizers, but the related resources have been deleted so the handler cannot find it. To eliminate the errors, we need to find and delete the resource that causes the error. See more information [here](https://github.com/rancher/rancher/issues/35050#issuecomment-937968556) diff --git a/content/rancher/v2.6/en/best-practices/_index.md b/content/rancher/v2.6/en/best-practices/_index.md index 4a9912da47b..37310b8fdd9 100644 --- a/content/rancher/v2.6/en/best-practices/_index.md +++ b/content/rancher/v2.6/en/best-practices/_index.md @@ -12,9 +12,7 @@ Use the navigation bar on the left to find the current best practices for managi For more guidance on best practices, you can consult these resources: - [Security]({{}}/rancher/v2.6/en/security/) -- [Rancher Blog](https://rancher.com/blog/) - - [Articles about best practices on the Rancher blog](https://rancher.com/tags/best-practices/) - - [101 More Security Best Practices for Kubernetes](https://rancher.com/blog/2019/2019-01-17-101-more-kubernetes-security-best-practices/) +- [Rancher Blog](https://www.suse.com/c/rancherblog/) - [Rancher Forum](https://forums.rancher.com/) - [Rancher Users Slack](https://slack.rancher.io/) - [Rancher Labs YouTube Channel - Online Meetups, Demos, Training, and Webinars](https://www.youtube.com/channel/UCh5Xtp82q8wjijP8npkVTBA/featured) diff --git a/content/rancher/v2.6/en/best-practices/rancher-server/deployment-types/_index.md b/content/rancher/v2.6/en/best-practices/rancher-server/deployment-types/_index.md index 2c9d5a218b1..35e0fcaf41a 100644 --- a/content/rancher/v2.6/en/best-practices/rancher-server/deployment-types/_index.md +++ b/content/rancher/v2.6/en/best-practices/rancher-server/deployment-types/_index.md @@ -15,7 +15,7 @@ When you set up your high-availability Rancher installation, consider the follow Don't run other workloads or microservices in the Kubernetes cluster that Rancher is installed on. ### Make sure nodes are configured correctly for Kubernetes ### -It's important to follow K8s and etcd best practices when deploying your nodes, including disabling swap, double checking you have full network connectivity between all machines in the cluster, using unique hostnames, MAC addresses, and product_uuids for every node, checking that all correct ports are opened, and deploying with ssd backed etcd. More details can be found in the [kubernetes docs](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#before-you-begin) and [etcd's performance op guide](https://github.com/etcd-io/etcd/blob/master/Documentation/op-guide/performance.md) +It's important to follow K8s and etcd best practices when deploying your nodes, including disabling swap, double checking you have full network connectivity between all machines in the cluster, using unique hostnames, MAC addresses, and product_uuids for every node, checking that all correct ports are opened, and deploying with ssd backed etcd. More details can be found in the [kubernetes docs](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#before-you-begin) and [etcd's performance op guide](https://etcd.io/docs/v3.4/op-guide/performance/). ### When using RKE: Back up the Statefile RKE keeps record of the cluster state in a file called `cluster.rkestate`. This file is important for the recovery of a cluster and/or the continued maintenance of the cluster through RKE. Because this file contains certificate material, we strongly recommend encrypting this file before backing up. After each run of `rke up` you should backup the state file. diff --git a/content/rancher/v2.6/en/best-practices/rancher-server/rancher-in-vsphere/_index.md b/content/rancher/v2.6/en/best-practices/rancher-server/rancher-in-vsphere/_index.md index 48d5459b691..207ab8accb2 100644 --- a/content/rancher/v2.6/en/best-practices/rancher-server/rancher-in-vsphere/_index.md +++ b/content/rancher/v2.6/en/best-practices/rancher-server/rancher-in-vsphere/_index.md @@ -50,7 +50,7 @@ https://rancher.com/docs/rancher/v2.6/en/installation/requirements/ ### Leverage VM Templates to Construct the Environment -To facilitate consistency across the deployed Virtual Machines across the environment, consider the use of "Golden Images" in the form of VM templates. Packer can be used to accomplish this, adding greater customisation options. +To facilitate the consistency of Virtual Machines deployed across the environment, consider the use of "Golden Images" in the form of VM templates. Packer can be used to accomplish this, adding greater customization options. ### Leverage DRS Anti-Affinity Rules (Where Possible) to Separate Rancher Cluster Nodes Across ESXi Hosts diff --git a/content/rancher/v2.6/en/cli/_index.md b/content/rancher/v2.6/en/cli/_index.md index 89928a7dc9b..553cb44c5c2 100644 --- a/content/rancher/v2.6/en/cli/_index.md +++ b/content/rancher/v2.6/en/cli/_index.md @@ -1,17 +1,33 @@ --- -title: Using the Rancher Command Line Interface -description: The Rancher CLI is a unified tool that you can use to interact with Rancher. With it, you can operate Rancher using a command line interface rather than the GUI -metaTitle: "Using the Rancher Command Line Interface " -metaDescription: "The Rancher CLI is a unified tool that you can use to interact with Rancher. With it, you can operate Rancher using a command line interface rather than the GUI" +title: CLI with Rancher +description: Interact with Rancher using command line interface (CLI) tools from your workstation. weight: 21 --- +- [Rancher CLI](#rancher-cli) + - [Download Rancher CLI](#download-rancher-cli) + - [Requirements](#requirements) + - [CLI Authentication](#cli-authentication) + - [Project Selection](#project-selection) + - [Commands](#commands) + - [Rancher CLI Help](#rancher-cli-help) + - [Limitations](#limitations) +- [kubectl](#kubectl) + - [kubectl Utility](#kubectl-utility) + - [Authentication with kubectl and kubeconfig Tokens with TTL](#authentication-with-kubectl-and-kubeconfig-tokens-with-ttl) + +# Rancher CLI + The Rancher CLI (Command Line Interface) is a unified tool that you can use to interact with Rancher. With this tool, you can operate Rancher using a command line rather than the GUI. ### Download Rancher CLI The binary can be downloaded directly from the UI. The link can be found in the right hand side of the footer in the UI. We have binaries for Windows, Mac, and Linux. You can also check the [releases page for our CLI](https://github.com/rancher/cli/releases) for direct downloads of the binary. +1. In the upper left corner, click **☰**. +1. At the bottom, click **v2.6.x**, where **v2.6.x** is a hyperlinked text indicating the installed Rancher version. +1. Under the **CLI Downloads section**, there are links to download the binaries for Windows, Mac, and Linux. You can also check the [releases page for our CLI](https://github.com/rancher/cli/releases) for direct downloads of the binary. + ### Requirements After you download the Rancher CLI, you need to make a few configurations. Rancher CLI requires: @@ -36,11 +52,11 @@ Before you can perform any commands, you must select a Rancher project to perfor **Example: `./rancher context switch` Output** ``` User:rancher-cli-directory user$ ./rancher context switch -NUMBER CLUSTER NAME PROJECT ID PROJECT NAME -1 cluster-2 c-7q96s:p-h4tmb project-2 -2 cluster-2 c-7q96s:project-j6z6d Default -3 cluster-1 c-lchzv:p-xbpdt project-1 -4 cluster-1 c-lchzv:project-s2mch Default +NUMBER CLUSTER NAME PROJECT ID PROJECT NAME +1 cluster-2 c-7q96s:p-h4tmb project-2 +2 cluster-2 c-7q96s:project-j6z6d Default +3 cluster-1 c-lchzv:p-xbpdt project-1 +4 cluster-1 c-lchzv:project-s2mch Default Select a Project: ``` @@ -51,13 +67,15 @@ INFO[0005] Setting new context to project project-1 INFO[0005] Saving config to /Users/markbishop/.rancher/cli2.json ``` +Ensure you can run `rancher kubectl get pods` successfully. + ### Commands The following commands are available for use in Rancher CLI. | Command | Result | |---|---| -| `apps, [app]` | Performs operations on catalog applications (i.e. individual [Helm charts](https://docs.helm.sh/developing_charts/) or Rancher charts. | +| `apps, [app]` | Performs operations on catalog applications (i.e., individual [Helm charts](https://docs.helm.sh/developing_charts/)) or Rancher charts. | | `catalog` | Performs operations on [catalogs]({{}}/rancher/v2.6/en/helm-charts/). | | `clusters, [cluster]` | Performs operations on your [clusters]({{}}/rancher/v2.6/en/cluster-provisioning/). | | `context` | Switches between Rancher [projects]({{}}/rancher/v2.6/en/cluster-admin/projects-and-namespaces/). For an example, see [Project Selection](#project-selection). | @@ -78,3 +96,38 @@ The following commands are available for use in Rancher CLI. Once logged into Rancher Server using the CLI, enter `./rancher --help` for a list of commands. All commands accept the `--help` flag, which documents each command's usage. + +### Limitations + +The Rancher CLI **cannot** be used to install [dashboard apps or Rancher feature charts](../helm-charts/). + +# kubectl + +Interact with Rancher using kubectl. + +### kubectl Utility + +Install the `kubectl` utility. See [install kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/). + +Configure kubectl by visiting your cluster in the Rancher Web UI, clicking on `Kubeconfig`, copying contents, and putting them into your `~/.kube/config` file. + +Run `kubectl cluster-info` or `kubectl get pods` successfully. + +### Authentication with kubectl and kubeconfig Tokens with TTL + +_Requirements_ + +If admins have [enforced TTL on kubeconfig tokens]({{}}/rancher/v2.6/en/api/api-tokens/#setting-ttl-on-kubeconfig-tokens), the kubeconfig file requires the [Rancher CLI](../cli) to be present in your PATH when you run `kubectl`. Otherwise, you’ll see an error like: +`Unable to connect to the server: getting credentials: exec: exec: "rancher": executable file not found in $PATH`. + +This feature enables kubectl to authenticate with the Rancher server and get a new kubeconfig token when required. The following auth providers are currently supported: + +1. Local +2. Active Directory (LDAP only) +3. FreeIPA +4. OpenLDAP +5. SAML providers: Ping, Okta, ADFS, Keycloak, Shibboleth + +When you first run kubectl, for example, `kubectl get pods`, it will ask you to pick an auth provider and log in with the Rancher server. +The kubeconfig token is cached in the path where you run kubectl under `./.cache/token`. This token is valid until [it expires](../../api/api-tokens/#setting-ttl-on-kubeconfig-tokens-period), or [gets deleted from the Rancher server](../../api/api-tokens/#deleting-tokens). +Upon expiration, the next `kubectl get pods` will ask you to log in with the Rancher server again. diff --git a/content/rancher/v2.6/en/cluster-admin/certificate-rotation/_index.md b/content/rancher/v2.6/en/cluster-admin/certificate-rotation/_index.md index 38eb69b8cb1..c38a4dd0d5b 100644 --- a/content/rancher/v2.6/en/cluster-admin/certificate-rotation/_index.md +++ b/content/rancher/v2.6/en/cluster-admin/certificate-rotation/_index.md @@ -9,12 +9,64 @@ By default, Kubernetes clusters require certificates and Rancher launched Kubern Certificates can be rotated for the following services: +{{% tabs %}} +{{% tab "RKE" %}} + - etcd -- kubelet +- kubelet (node certificate) +- kubelet (serving certificate, if [enabled]({{}}/rke/latest/en/config-options/services/#kubelet-options)) - kube-apiserver - kube-proxy - kube-scheduler - kube-controller-manager +{{% /tab %}} +{{% tab "RKE2" %}} + +- admin +- api-server +- controller-manager +- scheduler +- rke2-controller +- rke2-server +- cloud-controller +- etcd +- auth-proxy +- kubelet +- kube-proxy + +{{% /tab %}} +{{% /tabs %}} + > **Note:** For users who didn't rotate their webhook certificates, and they have expired after one year, please see this [page]({{}}/rancher/v2.6/en/troubleshooting/expired-webhook-certificates/) for help. + +### Certificate Rotation + +Rancher launched Kubernetes clusters have the ability to rotate the auto-generated certificates through the UI. + +1. In the upper left corner, click **☰ > Cluster Management**. +1. On the **Clusters** page, go to the cluster you want to rotate certificates for amd click **⋮ > Rotate Certificates**. +1. Select which certificates that you want to rotate. + + * Rotate all Service certificates (keep the same CA) + * Rotate an individual service and choose one of the services from the drop-down menu + +1. Click **Save**. + +**Results:** The selected certificates will be rotated and the related services will be restarted to start using the new certificate. + +### Additional Notes + +{{% tabs %}} +{{% tab "RKE" %}} + +Even though the RKE CLI can use custom certificates for the Kubernetes cluster components, Rancher currently doesn't allow the ability to upload these in Rancher launched Kubernetes clusters. + +{{% /tab %}} +{{% tab "RKE2" %}} + +In RKE2, both etcd and control plane nodes are treated as the same `server` concept. As such, when rotating certificates of services specific to either of these components will result in certificates being rotated on both. The certificates will only change for the specified service, but you will see nodes for both components go into an updating state. You may also see worker only nodes go into an updating state. This is to restart the workers after a certificate change to ensure they get the latest client certs. + +{{% /tab %}} +{{% /tabs %}} diff --git a/content/rancher/v2.6/en/cluster-admin/cluster-access/_index.md b/content/rancher/v2.6/en/cluster-admin/cluster-access/_index.md index 9d092a24550..291423864fc 100644 --- a/content/rancher/v2.6/en/cluster-admin/cluster-access/_index.md +++ b/content/rancher/v2.6/en/cluster-admin/cluster-access/_index.md @@ -20,7 +20,7 @@ Rancher provides an intuitive user interface for interacting with your clusters. You can use the Kubernetes command-line tool, [kubectl](https://kubernetes.io/docs/reference/kubectl/overview/), to manage your clusters. You have two options for using kubectl: -- **Rancher kubectl shell:** Interact with your clusters by launching a kubectl shell available in the Rancher UI. This option requires no configuration actions on your part. For more information, see [Accessing Clusters with kubectl Shell]({{}}/rancher/v2.6/en/cluster-access/kubectl/). +- **Rancher kubectl shell:** Interact with your clusters by launching a kubectl shell available in the Rancher UI. This option requires no configuration actions on your part. For more information, see [Accessing Clusters with kubectl Shell]({{}}/rancher/v2.6/en/cluster-admin/cluster-access/kubectl/). - **Terminal remote connection:** You can also interact with your clusters by installing [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/) on your local desktop and then copying the cluster's kubeconfig file to your local `~/.kube/config` directory. For more information, see [Accessing Clusters with kubectl and a kubeconfig File](./kubectl/). ### Rancher CLI diff --git a/content/rancher/v2.6/en/cluster-admin/editing-clusters/_index.md b/content/rancher/v2.6/en/cluster-admin/editing-clusters/_index.md index c9e257784be..cab7cdd0d06 100644 --- a/content/rancher/v2.6/en/cluster-admin/editing-clusters/_index.md +++ b/content/rancher/v2.6/en/cluster-admin/editing-clusters/_index.md @@ -12,8 +12,8 @@ For information on editing cluster membership, go to [this page.]({{}}/ The cluster configuration options depend on the type of Kubernetes cluster: - [RKE Cluster Configuration](./rke-config-reference) -- [RKE2 Cluster Configuration](./rke2-config-reference) (Tech Preview) -- [K3s Cluster Configuration](./k3s-config-reference) (Tech Preview) +- [RKE2 Cluster Configuration](./rke2-config-reference) +- [K3s Cluster Configuration](./k3s-config-reference) - [EKS Cluster Configuration](./eks-config-reference) - [GKE Cluster Configuration](./gke-config-reference) - [AKS Cluster Configuration](./aks-config-reference) diff --git a/content/rancher/v2.6/en/cluster-admin/editing-clusters/eks-config-reference/_index.md b/content/rancher/v2.6/en/cluster-admin/editing-clusters/eks-config-reference/_index.md index 67d3e46d94f..4eec75f21f7 100644 --- a/content/rancher/v2.6/en/cluster-admin/editing-clusters/eks-config-reference/_index.md +++ b/content/rancher/v2.6/en/cluster-admin/editing-clusters/eks-config-reference/_index.md @@ -11,7 +11,7 @@ Complete each drop-down and field using the information obtained for your IAM po | Setting | Description | | ---------- | -------------------------------------------------------------------------------------------------------------------- | | Region | From the drop-down choose the geographical region in which to build your cluster. | -| Cloud Credentials | Select the cloud credentials that you created for your IAM policy. For more information on creating cloud credentials in Rancher, refer to [this page.]({{}}/rancher/v2.x/en/user-settings/cloud-credentials/) | +| Cloud Credentials | Select the cloud credentials that you created for your IAM policy. For more information on creating cloud credentials in Rancher, refer to [this page.]({{}}/rancher/v2.6/en/user-settings/cloud-credentials/) | ### Service Role diff --git a/content/rancher/v2.6/en/cluster-admin/editing-clusters/gke-config-reference/_index.md b/content/rancher/v2.6/en/cluster-admin/editing-clusters/gke-config-reference/_index.md index 30b22867584..b87543a3157 100644 --- a/content/rancher/v2.6/en/cluster-admin/editing-clusters/gke-config-reference/_index.md +++ b/content/rancher/v2.6/en/cluster-admin/editing-clusters/gke-config-reference/_index.md @@ -103,13 +103,13 @@ The address range assigned to the services in the cluster. Must be a valid CIDR _Mutable: no_ -> Warning: private clusters require additional planning and configuration outside of Rancher. Refer to the [private cluster guide]({{< baseurl >}}/rancher/v2.6/en/cluster-provisioning/hosted-kubernetes-clusters/gke/private-clusters/). +> Warning: private clusters require additional planning and configuration outside of Rancher. Refer to the [private cluster guide]({{< baseurl >}}/rancher/v2.6/en/cluster-admin/editing-clusters/gke-config-reference/private-clusters/). Assign nodes only internal IP addresses. Private cluster nodes cannot access the public internet unless additional networking steps are taken in GCP. ### Enable Private Endpoint -> Warning: private clusters require additional planning and configuration outside of Rancher. Refer to the [private cluster guide]({{< baseurl >}}/rancher/v2.6/en/cluster-provisioning/hosted-kubernetes-clusters/gke/#private-clusters). +> Warning: private clusters require additional planning and configuration outside of Rancher. Refer to the [private cluster guide]({{< baseurl >}}/rancher/v2.6/en/cluster-admin/editing-clusters/gke-config-reference/private-clusters/). _Mutable: no_ diff --git a/content/rancher/v2.6/en/cluster-admin/editing-clusters/gke-config-reference/private-clusters/_index.md b/content/rancher/v2.6/en/cluster-admin/editing-clusters/gke-config-reference/private-clusters/_index.md index cf3b3963f05..95d9bd39bcc 100644 --- a/content/rancher/v2.6/en/cluster-admin/editing-clusters/gke-config-reference/private-clusters/_index.md +++ b/content/rancher/v2.6/en/cluster-admin/editing-clusters/gke-config-reference/private-clusters/_index.md @@ -21,7 +21,7 @@ If restricting outgoing internet access is not a concern for your organization, >**Note** >This scenario is not officially supported, but is described for cases in which using the Cloud NAT service is not sufficient. -If restricting both incoming and outgoing traffic to nodes is a requirement, follow the air-gapped installation instructions to set up a private container image [registry](https://rancher.com/docs/rancher/v2.x/en/installation/other-installation-methods/air-gap/) on the VPC where the cluster is going to be, allowing the cluster nodes to access and download the images they need to run the cluster agent. If the control plane endpoint is also private, Rancher will need [direct access](#direct-access) to it. +If restricting both incoming and outgoing traffic to nodes is a requirement, follow the air-gapped installation instructions to set up a private container image [registry](https://rancher.com/docs/rancher/v2.6/en/installation/other-installation-methods/air-gap/) on the VPC where the cluster is going to be, allowing the cluster nodes to access and download the images they need to run the cluster agent. If the control plane endpoint is also private, Rancher will need [direct access](#direct-access) to it. ### Private Control Plane Endpoint diff --git a/content/rancher/v2.6/en/cluster-admin/editing-clusters/k3s-config-reference/_index.md b/content/rancher/v2.6/en/cluster-admin/editing-clusters/k3s-config-reference/_index.md index 47ee18b44e2..00d784cfa0c 100644 --- a/content/rancher/v2.6/en/cluster-admin/editing-clusters/k3s-config-reference/_index.md +++ b/content/rancher/v2.6/en/cluster-admin/editing-clusters/k3s-config-reference/_index.md @@ -1,5 +1,5 @@ --- -title: K3s Cluster Configuration Reference (Tech Preview) +title: K3s Cluster Configuration Reference shortTitle: K3s Cluster Configuration weight: 6 --- diff --git a/content/rancher/v2.6/en/cluster-admin/editing-clusters/rke-config-reference/_index.md b/content/rancher/v2.6/en/cluster-admin/editing-clusters/rke-config-reference/_index.md index 2b530a8db5e..27e73a4c408 100644 --- a/content/rancher/v2.6/en/cluster-admin/editing-clusters/rke-config-reference/_index.md +++ b/content/rancher/v2.6/en/cluster-admin/editing-clusters/rke-config-reference/_index.md @@ -189,7 +189,7 @@ Option to enable or disable [recurring etcd snapshots]({{}}/rke/latest/ ### Agent Environment Variables -Option to set environment variables for [rancher agents]({{}}/rancher/v2.x/en/cluster-provisioning/rke-clusters/rancher-agents/). The environment variables can be set using key value pairs. If rancher agent requires use of proxy to communicate with Rancher server, `HTTP_PROXY`, `HTTPS_PROXY` and `NO_PROXY` environment variables can be set using agent environment variables. +Option to set environment variables for [rancher agents]({{}}/rancher/v2.6/en/cluster-provisioning/rke-clusters/rancher-agents/). The environment variables can be set using key value pairs. If rancher agent requires use of proxy to communicate with Rancher server, `HTTP_PROXY`, `HTTPS_PROXY` and `NO_PROXY` environment variables can be set using agent environment variables. ### Updating ingress-nginx @@ -341,7 +341,10 @@ Example: local_cluster_auth_endpoint: enabled: true fqdn: "FQDN" - ca_certs: "BASE64_CACERT" + ca_certs: |- + -----BEGIN CERTIFICATE----- + ... + -----END CERTIFICATE----- ``` ### Custom Network Plug-in diff --git a/content/rancher/v2.6/en/cluster-admin/editing-clusters/rke2-config-reference/_index.md b/content/rancher/v2.6/en/cluster-admin/editing-clusters/rke2-config-reference/_index.md index 907fbfb3aea..3e1ee54d5db 100644 --- a/content/rancher/v2.6/en/cluster-admin/editing-clusters/rke2-config-reference/_index.md +++ b/content/rancher/v2.6/en/cluster-admin/editing-clusters/rke2-config-reference/_index.md @@ -1,5 +1,5 @@ --- -title: RKE2 Cluster Configuration Reference (Tech Preview) +title: RKE2 Cluster Configuration Reference shortTitle: RKE2 Cluster Configuration weight: 5 --- @@ -8,7 +8,7 @@ This section covers the configuration options that are available in Rancher for # Overview -You can configure the Kubernetes options one of two ways: +You can configure the Kubernetes options in one of the two following ways: - [Rancher UI](#configuration-options-in-the-rancher-ui): Use the Rancher UI to select options that are commonly customized when setting up a Kubernetes cluster. - [Cluster Config File](#cluster-config-file): Instead of using the Rancher UI to choose Kubernetes options for the cluster, advanced users can create an RKE2 config file. Using a config file allows you to set any of the [options](https://docs.rke2.io/install/install_options/install_options) available in an RKE2 installation. @@ -28,17 +28,27 @@ For more detail, see [Upgrading Kubernetes]({{}}/rancher/v2.6/en/cluste The [Network Provider](https://kubernetes.io/docs/concepts/cluster-administration/networking/) that the cluster uses. -> After you launch the cluster, you cannot change your network provider. Therefore, choose which network provider you want to use carefully, as Kubernetes doesn't allow switching between network providers. Once a cluster is created with a network provider, changing network providers would require you tear down the entire cluster and all its applications. +> After you launch the cluster, you cannot change your network provider. Therefore, choose which network provider you want to use carefully, as Kubernetes doesn't allow switching between network providers. Once a cluster is created with a network provider, changing network providers would require you to tear down the entire cluster and all its applications. Out of the box, Rancher is compatible with the following network providers: - [Canal](https://github.com/projectcalico/canal) -- [Cilium](https://cilium.io/) +- [Cilium](https://cilium.io/)* - [Calico](https://docs.projectcalico.org/v3.11/introduction/) - [Multus](https://github.com/k8snetworkplumbingwg/multus-cni) +\* When using [project network isolation](#project-network-isolation) in the [Cilium CNI]({{}}/rancher/v2.6/en/faq/networking/cni-providers/#cilium), it is possible to enable cross-node ingress routing. Click the [CNI provider docs]({{}}/rancher/v2.6/en/faq/networking/cni-providers/#ingress-routing-across-nodes-in-cilium) to learn more. + For more details on the different networking providers and how to configure them, please view our [RKE2 documentation](https://docs.rke2.io/install/network_options/). +##### Dual-stack Networking + +[Dual-stack](https://docs.rke2.io/install/network_options/#dual-stack-configuration) networking is supported for all CNI providers. To configure RKE2 in dual-stack mode, set valid IPv4/IPv6 CIDRs for your [Cluster CIDR](#cluster-cidr) and/or [Service CIDR](#service-cidr). + +###### Additional Configuration {#dual-stack-additional-config} + +When using `cilium` or `multus,cilium` as your container network interface provider, ensure the **Enable IPv6 Support** option is also enabled. + #### Cloud Provider You can configure a [Kubernetes cloud provider]({{}}/rancher/v2.6/en/cluster-provisioning/rke-clusters/cloud-providers). If you want to use dynamically provisioned [volumes and storage]({{}}/rancher/v2.6/en/cluster-admin/volumes-and-storage/) in Kubernetes, typically you must select the specific cloud provider in order to use it. For example, if you want to use Amazon EBS, you would need to select the `aws` cloud provider. @@ -81,7 +91,7 @@ Each cloud provider capable of launching a cluster using RKE2 can collect metric ### Add-On Config -Additional Kubernetes manifests, managed as a [Add-on](https://kubernetes.io/docs/concepts/cluster-administration/addons/), to apply to the cluster on startup. Refer to the [RKE2 documentation](https://docs.rke2.io/helm/#automatically-deploying-manifests-and-helm-charts) for details. +Additional Kubernetes manifests, managed as an [Add-on](https://kubernetes.io/docs/concepts/cluster-administration/addons/), to apply to the cluster on startup. Refer to the [RKE2 documentation](https://docs.rke2.io/helm/#automatically-deploying-manifests-and-helm-charts) for details. ### Agent Environment Vars @@ -101,12 +111,24 @@ Option to choose whether to expose etcd metrics to the public or only within the #### Cluster CIDR -IPv4/IPv6 network CIDRs to use for pod IPs (default: 10.42.0.0/16). +IPv4 and/or IPv6 network CIDRs to use for pod IPs (default: 10.42.0.0/16). + +##### Dual-stack Networking + +To configure [dual-stack](https://docs.rke2.io/install/network_options/#dual-stack-configuration) mode, enter a valid IPv4/IPv6 CIDR. For example `10.42.0.0/16,2001:cafe:42:0::/56`. + +[Additional configuration](#dual-stack-additional-config) is required when using `cilium` or `multus,cilium` as your [container network](#container-network) interface provider. #### Service CIDR IPv4/IPv6 network CIDRs to use for service IPs (default: 10.43.0.0/16). +##### Dual-stack Networking + +To configure [dual-stack](https://docs.rke2.io/install/network_options/#dual-stack-configuration) mode, enter a valid IPv4/IPv6 CIDR. For example `10.42.0.0/16,2001:cafe:42:0::/56`. + +[Additional configuration](#dual-stack-additional-config) is required when using `cilium ` or `multus,cilium` as your [container network](#container-network) interface provider. + #### Cluster DNS IPv4 Cluster IP for coredns service. Should be in your service-cidr range (default: 10.43.0.10). @@ -139,7 +161,7 @@ Select the image repository to pull Rancher images from. For more details and co ### Upgrade Strategy -#### Controle Plane Concurrency +#### Control Plane Concurrency Select how many nodes can be upgraded at the same time. Can be a fixed number or percentage. @@ -164,5 +186,3 @@ Option to set kubelet options for different nodes. For available options, refer Instead of using the Rancher UI forms to choose Kubernetes options for the cluster, advanced users can create an RKE2 config file. Using a config file allows you to set any of the [options](https://docs.rke2.io/install/install_options/install_options) available in an RKE2 installation. To edit an RKE2 config file directly from the Rancher UI, click **Edit as YAML**. - - diff --git a/content/rancher/v2.6/en/cluster-admin/nodes/_index.md b/content/rancher/v2.6/en/cluster-admin/nodes/_index.md index a60fb2d562a..9f2243cceab 100644 --- a/content/rancher/v2.6/en/cluster-admin/nodes/_index.md +++ b/content/rancher/v2.6/en/cluster-admin/nodes/_index.md @@ -40,11 +40,11 @@ The following table lists which node options are available for each type of clus | [Download Keys](#ssh-into-a-node-hosted-by-an-infrastructure-provider) | ✓ | | | | | Download SSH key in order to SSH into the node. | | [Node Scaling](#scaling-nodes) | ✓ | | | ✓ | | Scale the number of nodes in the node pool up or down. | -[1]: {{}}/rancher/v2.x/en/cluster-provisioning/rke-clusters/node-pools/ -[2]: {{}}/rancher/v2.x/en/cluster-provisioning/rke-clusters/custom-nodes/ -[3]: {{}}/rancher/v2.x/en/cluster-provisioning/hosted-kubernetes-clusters/ -[4]: {{}}/rancher/v2.x/en/cluster-provisioning/registered-clusters/ -[5]: {{}}/rancher/v2.x/en/cluster-provisioning/registered-clusters/ +[1]: {{}}/rancher/v2.6/en/cluster-provisioning/rke-clusters/node-pools/ +[2]: {{}}/rancher/v2.6/en/cluster-provisioning/rke-clusters/custom-nodes/ +[3]: {{}}/rancher/v2.6/en/cluster-provisioning/hosted-kubernetes-clusters/ +[4]: {{}}/rancher/v2.6/en/cluster-provisioning/registered-clusters/ +[5]: {{}}/rancher/v2.6/en/cluster-provisioning/registered-clusters/ \* Delete option accessible via View API diff --git a/content/rancher/v2.6/en/cluster-admin/projects-and-namespaces/_index.md b/content/rancher/v2.6/en/cluster-admin/projects-and-namespaces/_index.md index 71820214033..d1ae03863b4 100644 --- a/content/rancher/v2.6/en/cluster-admin/projects-and-namespaces/_index.md +++ b/content/rancher/v2.6/en/cluster-admin/projects-and-namespaces/_index.md @@ -8,6 +8,8 @@ A namespace is a Kubernetes concept that allows a virtual cluster within a clust A project is a group of namespaces, and it is a concept introduced by Rancher. Projects allow you to manage multiple namespaces as a group and perform Kubernetes operations in them. You can use projects to support multi-tenancy, so that a team can access a project within a cluster without having access to other projects in the same cluster. +>**Note:** As of Rancher v2.6, projects are de-emphasized on the UI because it is no longer required to create any Kubernetes resources within a project scope. However, resources such as [Secrets]({{}}/rancher/v2.6/en/k8s-in-rancher/secrets/#creating-secrets-in-projects) can still be created in a project scope if the legacy feature flag is enabled. + This section describes how projects and namespaces work with Rancher. It covers the following topics: - [About namespaces](#about-namespaces) @@ -150,6 +152,9 @@ By default, your user is added as the project `Owner`. >**Notes on Permissions:** > >- Users assigned the `Owner` or `Member` role for a project automatically inherit the `namespace creation` role. However, this role is a [Kubernetes ClusterRole](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#role-and-clusterrole), meaning its scope extends to all projects in the cluster. Therefore, users explicitly assigned the `Owner` or `Member` role for a project can create namespaces in other projects they're assigned to, even with only the `Read Only` role assigned. +> +>- By default, the Rancher role of `project-member` inherits from the `Kubernetes-edit` role, and the `project-owner` role inherits from the `Kubernetes-admin` role. As such, both `project-member` and `project-owner` roles will allow for namespace management, including the ability to create and delete namespaces. +> >- Choose `Custom` to create a custom role on the fly: [Custom Project Roles]({{}}/rancher/v2.6/en/admin-settings/rbac/cluster-project-roles/#custom-project-roles). To add members: diff --git a/content/rancher/v2.6/en/cluster-admin/volumes-and-storage/_index.md b/content/rancher/v2.6/en/cluster-admin/volumes-and-storage/_index.md index 9467dfa660f..745fbce0131 100644 --- a/content/rancher/v2.6/en/cluster-admin/volumes-and-storage/_index.md +++ b/content/rancher/v2.6/en/cluster-admin/volumes-and-storage/_index.md @@ -44,7 +44,7 @@ For details and prerequisites, refer to [this page.](./provisioning-new-storage) Longhorn is free, open source software. Originally developed by Rancher Labs, it is now being developed as a sandbox project of the Cloud Native Computing Foundation. It can be installed on any Kubernetes cluster with Helm, with kubectl, or with the Rancher UI. -If you have a pool of block storage, Longhorn can help you provide persistent storage to your Kubernetes cluster without relying on cloud providers. For more information about Longhorn features, refer to the [documentation.](https://longhorn.io/docs/1.0.2/what-is-longhorn/) +If you have a pool of block storage, Longhorn can help you provide persistent storage to your Kubernetes cluster without relying on cloud providers. For more information about Longhorn features, refer to the [documentation.](https://longhorn.io/docs/latest/what-is-longhorn/) Rancher v2.5 simplified the process of installing Longhorn on a Rancher-managed cluster. For more information, see [this page.]({{}}/rancher/v2.6/en/longhorn) diff --git a/content/rancher/v2.6/en/cluster-admin/volumes-and-storage/iscsi-volumes/_index.md b/content/rancher/v2.6/en/cluster-admin/volumes-and-storage/iscsi-volumes/_index.md index 01d87aabd59..c8a14cb458d 100644 --- a/content/rancher/v2.6/en/cluster-admin/volumes-and-storage/iscsi-volumes/_index.md +++ b/content/rancher/v2.6/en/cluster-admin/volumes-and-storage/iscsi-volumes/_index.md @@ -17,9 +17,12 @@ If you encounter this issue, you can work around it by installing the initiator After installing the initiator tool on your nodes, edit the YAML for your cluster, editing the kubelet configuration to mount the iSCSI binary and configuration, as shown in the sample below. ->**Note:** +>**Notes:** +> +>- Before updating your Kubernetes YAML to mount the iSCSI binary and configuration, make sure either the `open-iscsi` (deb) or `iscsi-initiator-utils` (yum) package is installed on your cluster nodes. If this package isn't installed _before_ the bind mounts are created in your Kubernetes YAML, Docker will automatically create the directories and files on each node and will not allow the package install to succeed.
+>
> ->Before updating your Kubernetes YAML to mount the iSCSI binary and configuration, make sure either the `open-iscsi` (deb) or `iscsi-initiator-utils` (yum) package is installed on your cluster nodes. If this package isn't installed _before_ the bind mounts are created in your Kubernetes YAML, Docker will automatically create the directories and files on each node and will not allow the package install to succeed. +>- The example YAML below does not apply to K3s, but only to RKE clusters. Since the K3s kubelet does not run in a container, adding extra binds is not necessary. However, all iSCSI tools must still be installed on your K3s nodes. ``` services: diff --git a/content/rancher/v2.6/en/cluster-provisioning/_index.md b/content/rancher/v2.6/en/cluster-provisioning/_index.md index 5a33cad88f8..9e9f44c4c87 100644 --- a/content/rancher/v2.6/en/cluster-provisioning/_index.md +++ b/content/rancher/v2.6/en/cluster-provisioning/_index.md @@ -20,6 +20,7 @@ This section covers the following topics: - [Launching Kubernetes and Provisioning Nodes in an Infrastructure Provider](#launching-kubernetes-and-provisioning-nodes-in-an-infrastructure-provider) - [Launching Kubernetes on Existing Custom Nodes](#launching-kubernetes-on-existing-custom-nodes) - [Registering Existing Clusters](#registering-existing-clusters) +- [Programmatically Creating Clusters](#programmatically-creating-clusters) @@ -78,3 +79,9 @@ Registering EKS clusters now provides additional benefits. For the most part, re When you delete an EKS cluster that was created in Rancher, the cluster is destroyed. When you delete an EKS cluster that was registered in Rancher, it is disconnected from the Rancher server, but it still exists and you can still access it in the same way you did before it was registered in Rancher. For more information, see [this page.](./registered-clusters) + +# Programmatically Creating Clusters + +The most common way to programmatically deploy Kubernetes clusters through Rancher is by using the Rancher2 Terraform provider. The documentation for creating clusters with Terraform is [here.](https://registry.terraform.io/providers/rancher/rancher2/latest/docs/resources/cluster) + +EKS, GKE, AKS clusters and RKE clusters can be created or imported with Terraform. \ No newline at end of file diff --git a/content/rancher/v2.6/en/cluster-provisioning/hosted-kubernetes-clusters/aks/_index.md b/content/rancher/v2.6/en/cluster-provisioning/hosted-kubernetes-clusters/aks/_index.md index e3aa18629c8..f54e1e42bdd 100644 --- a/content/rancher/v2.6/en/cluster-provisioning/hosted-kubernetes-clusters/aks/_index.md +++ b/content/rancher/v2.6/en/cluster-provisioning/hosted-kubernetes-clusters/aks/_index.md @@ -14,8 +14,8 @@ You can use Rancher to create a cluster hosted in Microsoft Azure Kubernetes Ser - [Role-based Access Control](#role-based-access-control) - [AKS Cluster Configuration Reference](#aks-cluster-configuration-reference) - [Private Clusters](#private-clusters) -- [Minimum AKS Permissions](#minimum-aks-permissions) - [Syncing](#syncing) +- [Programmatically Creating AKS Clusters](#programmatically-creating-aks-clusters) # Prerequisites in Microsoft Azure @@ -152,3 +152,7 @@ For more information about connecting to an AKS private cluster, see the [AKS do The AKS provisioner can synchronize the state of an AKS cluster between Rancher and the provider. For an in-depth technical explanation of how this works, see [Syncing.]({{}}/rancher/v2.6/en/cluster-admin/editing-clusters/syncing) For information on configuring the refresh interval, see [this section.]({{}}/rancher/v2.6/en/cluster-admin/editing-clusters/gke-config-reference/#configuring-the-refresh-interval) + +# Programmatically Creating AKS Clusters + +The most common way to programmatically deploy AKS clusters through Rancher is by using the Rancher2 Terraform provider. The documentation for creating clusters with Terraform is [here.](https://registry.terraform.io/providers/rancher/rancher2/latest/docs/resources/cluster) \ No newline at end of file diff --git a/content/rancher/v2.6/en/cluster-provisioning/hosted-kubernetes-clusters/eks/_index.md b/content/rancher/v2.6/en/cluster-provisioning/hosted-kubernetes-clusters/eks/_index.md index 3235e46801c..3f55612fbdd 100644 --- a/content/rancher/v2.6/en/cluster-provisioning/hosted-kubernetes-clusters/eks/_index.md +++ b/content/rancher/v2.6/en/cluster-provisioning/hosted-kubernetes-clusters/eks/_index.md @@ -17,6 +17,7 @@ Amazon EKS provides a managed control plane for your Kubernetes cluster. Amazon - [Minimum EKS Permissions](#minimum-eks-permissions) - [Syncing](#syncing) - [Troubleshooting](#troubleshooting) +- [Programmatically Creating EKS Clusters](#programmatically-creating-eks-clusters) # Prerequisites in Amazon Web Services >**Note** @@ -108,4 +109,8 @@ If your changes were overwritten, it could be due to the way the cluster data is If an unauthorized error is returned while attempting to modify or register the cluster and the cluster was not created with the role or user that your credentials belong to, refer to [Security and Compliance.](#security-and-compliance) -For any issues or troubleshooting details for your Amazon EKS Kubernetes cluster, please see this [documentation](https://docs.aws.amazon.com/eks/latest/userguide/troubleshooting.html). \ No newline at end of file +For any issues or troubleshooting details for your Amazon EKS Kubernetes cluster, please see this [documentation](https://docs.aws.amazon.com/eks/latest/userguide/troubleshooting.html). + +# Programmatically Creating EKS Clusters + +The most common way to programmatically deploy EKS clusters through Rancher is by using the Rancher2 Terraform provider. The documentation for creating clusters with Terraform is [here.](https://registry.terraform.io/providers/rancher/rancher2/latest/docs/resources/cluster) \ No newline at end of file diff --git a/content/rancher/v2.6/en/cluster-provisioning/hosted-kubernetes-clusters/eks/permissions/_index.md b/content/rancher/v2.6/en/cluster-provisioning/hosted-kubernetes-clusters/eks/permissions/_index.md index 0567e110d02..8f4539e1c6f 100644 --- a/content/rancher/v2.6/en/cluster-provisioning/hosted-kubernetes-clusters/eks/permissions/_index.md +++ b/content/rancher/v2.6/en/cluster-provisioning/hosted-kubernetes-clusters/eks/permissions/_index.md @@ -24,6 +24,7 @@ Resource targeting uses `*` as the ARN of many of the resources created cannot b "ec2:RunInstances", "ec2:RevokeSecurityGroupIngress", "ec2:RevokeSecurityGroupEgress", + "ec2:DescribeRegions", "ec2:DescribeVpcs", "ec2:DescribeTags", "ec2:DescribeSubnets", @@ -123,31 +124,6 @@ Resource targeting uses `*` as the ARN of many of the resources created cannot b ### Service Role Permissions -Rancher will create a service role with the following trust policy: - -```json -{ - "Version": "2012-10-17", - "Statement": [ - { - "Action": "sts:AssumeRole", - "Principal": { - "Service": "eks.amazonaws.com" - }, - "Effect": "Allow", - "Sid": "" - } - ] -} -``` - -This role will also have two role policy attachments with the following policies ARNs: - -``` -arn:aws:iam::aws:policy/AmazonEKSClusterPolicy -arn:aws:iam::aws:policy/AmazonEKSServicePolicy -``` - Permissions required for Rancher to create service role on users behalf during the EKS cluster creation process. ```json @@ -182,36 +158,66 @@ Permissions required for Rancher to create service role on users behalf during t } ``` +When an EKS cluster is created, Rancher will create a service role with the following trust policy: + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Action": "sts:AssumeRole", + "Principal": { + "Service": "eks.amazonaws.com" + }, + "Effect": "Allow", + "Sid": "" + } + ] +} +``` + +This role will also have two role policy attachments with the following policies ARNs: + +``` +arn:aws:iam::aws:policy/AmazonEKSClusterPolicy +arn:aws:iam::aws:policy/AmazonEKSServicePolicy +``` + ### VPC Permissions Permissions required for Rancher to create VPC and associated resources. ```json { - "Sid": "VPCPermissions", - "Effect": "Allow", - "Action": [ - "ec2:ReplaceRoute", - "ec2:ModifyVpcAttribute", - "ec2:ModifySubnetAttribute", - "ec2:DisassociateRouteTable", - "ec2:DetachInternetGateway", - "ec2:DescribeVpcs", - "ec2:DeleteVpc", - "ec2:DeleteTags", - "ec2:DeleteSubnet", - "ec2:DeleteRouteTable", - "ec2:DeleteRoute", - "ec2:DeleteInternetGateway", - "ec2:CreateVpc", - "ec2:CreateSubnet", - "ec2:CreateSecurityGroup", - "ec2:CreateRouteTable", - "ec2:CreateRoute", - "ec2:CreateInternetGateway", - "ec2:AttachInternetGateway", - "ec2:AssociateRouteTable" - ], - "Resource": "*" + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "VPCPermissions", + "Effect": "Allow", + "Action": [ + "ec2:ReplaceRoute", + "ec2:ModifyVpcAttribute", + "ec2:ModifySubnetAttribute", + "ec2:DisassociateRouteTable", + "ec2:DetachInternetGateway", + "ec2:DescribeVpcs", + "ec2:DeleteVpc", + "ec2:DeleteTags", + "ec2:DeleteSubnet", + "ec2:DeleteRouteTable", + "ec2:DeleteRoute", + "ec2:DeleteInternetGateway", + "ec2:CreateVpc", + "ec2:CreateSubnet", + "ec2:CreateSecurityGroup", + "ec2:CreateRouteTable", + "ec2:CreateRoute", + "ec2:CreateInternetGateway", + "ec2:AttachInternetGateway", + "ec2:AssociateRouteTable" + ], + "Resource": "*" + } + ] } -``` \ No newline at end of file +``` diff --git a/content/rancher/v2.6/en/cluster-provisioning/hosted-kubernetes-clusters/gke/_index.md b/content/rancher/v2.6/en/cluster-provisioning/hosted-kubernetes-clusters/gke/_index.md index 0f1d1eec0e1..95567faa8c9 100644 --- a/content/rancher/v2.6/en/cluster-provisioning/hosted-kubernetes-clusters/gke/_index.md +++ b/content/rancher/v2.6/en/cluster-provisioning/hosted-kubernetes-clusters/gke/_index.md @@ -10,6 +10,7 @@ weight: 2105 - [Configuration Reference](#configuration-reference) - [Updating Kubernetes Version](#updating-kubernetes-version) - [Syncing](#syncing) +- [Programmatically Creating GKE Clusters](#programmatically-creating-gke-clusters) # Prerequisites @@ -98,3 +99,6 @@ The GKE provisioner can synchronize the state of a GKE cluster between Rancher a For information on configuring the refresh interval, see [this section.]({{}}/rancher/v2.6/en/cluster-admin/editing-clusters/gke-config-reference/#configuring-the-refresh-interval) +# Programmatically Creating GKE Clusters + +The most common way to programmatically deploy GKE clusters through Rancher is by using the Rancher2 Terraform provider. The documentation for creating clusters with Terraform is [here.](https://registry.terraform.io/providers/rancher/rancher2/latest/docs/resources/cluster) \ No newline at end of file diff --git a/content/rancher/v2.6/en/cluster-provisioning/node-requirements/_index.md b/content/rancher/v2.6/en/cluster-provisioning/node-requirements/_index.md index bd24c9adeb8..519e8c31077 100644 --- a/content/rancher/v2.6/en/cluster-provisioning/node-requirements/_index.md +++ b/content/rancher/v2.6/en/cluster-provisioning/node-requirements/_index.md @@ -30,6 +30,13 @@ For information on how to install Docker, refer to the official [Docker document Some distributions of Linux derived from RHEL, including Oracle Linux, may have default firewall rules that block communication with Helm. We recommend disabling firewalld. For Kubernetes 1.19, firewalld must be turned off. +>**Note:** In RHEL 8.4, two extra services are included on the NetworkManager: `nm-cloud-setup.service` and `nm-cloud-setup.timer`. These services add a routing table that interferes with the CNI plugin's configuration. If these services are enabled, you must disable them using the command below, and then reboot the node to restore connectivity: +> +> ``` + systemctl disable nm-cloud-setup.service nm-cloud-setup.timer + reboot + ``` + ### SUSE Linux Nodes SUSE Linux may have a firewall that blocks all ports by default. In that situation, follow [these steps]({{}}/rancher/v2.6/en/installation/requirements/ports/#opening-suse-linux-ports) to open the ports needed for adding a host to a custom cluster. diff --git a/content/rancher/v2.6/en/cluster-provisioning/registered-clusters/_index.md b/content/rancher/v2.6/en/cluster-provisioning/registered-clusters/_index.md index 6ad807cd9fa..5bcc0cc73f0 100644 --- a/content/rancher/v2.6/en/cluster-provisioning/registered-clusters/_index.md +++ b/content/rancher/v2.6/en/cluster-provisioning/registered-clusters/_index.md @@ -12,10 +12,19 @@ The control that Rancher has to manage a registered cluster depends on the type - [Management Capabilities for Registered Clusters](#management-capabilities-for-registered-clusters) - [Configuring K3s Cluster Upgrades](#configuring-k3s-cluster-upgrades) - [Debug Logging and Troubleshooting for Registered K3s Clusters](#debug-logging-and-troubleshooting-for-registered-k3s-clusters) +- [Authorized Cluster Endpoint Support for RKE2 and K3s Clusters](#authorized-cluster-endpoint-support-for-rke2-and-k3s-clusters) - [Annotating Registered Clusters](#annotating-registered-clusters) # Prerequisites +### Kubernetes Node Roles + +Registered RKE Kubernetes clusters must have all three node roles - etcd, controlplane and worker. A cluster with only controlplane components cannot be registered in Rancher. + +For more information on RKE node roles, see the [best practices.]({{}}/rancher/v2.6/en/cluster-provisioning/production/#cluster-architecture) + +### Permissions + If your existing Kubernetes cluster already has a `cluster-admin` role defined, you must have this `cluster-admin` privilege to register the cluster in Rancher. In order to apply the privilege, you need to run: @@ -32,11 +41,14 @@ By default, GKE users are not given this privilege, so you will need to run the If you are registering a K3s cluster, make sure the `cluster.yml` is readable. It is protected by default. For details, refer to [Configuring a K3s cluster to enable importation to Rancher.](#configuring-a-k3s-cluster-to-enable-registration-in-rancher) +### EKS Clusters + +EKS clusters must have at least one managed node group to be imported into Rancher or provisioned from Rancher successfully. + # Registering a Cluster 1. Click **☰ > Cluster Management**. 1. On the **Clusters** page, **Import Existing**. -1. Enter a **Cluster Name**. 1. Choose the type of cluster. 1. Use **Member Roles** to configure user authorization for the cluster. Click **Add Member** to add users that can access the cluster. Use the **Role** drop-down to set permissions for each user. 1. If you are importing a generic Kubernetes cluster in Rancher, perform the following steps for setup:
@@ -48,7 +60,6 @@ If you are registering a K3s cluster, make sure the `cluster.yml` is readable. I 1. If you are using self-signed certificates, you will receive the message `certificate signed by unknown authority`. To work around this validation, copy the command starting with `curl` displayed in Rancher to your clipboard. Then run the command on a node where kubeconfig is configured to point to the cluster you want to import. 1. When you finish running the command(s) on your node, click **Done**. - **Result:** - Your cluster is registered and assigned a state of **Pending**. Rancher is deploying resources to manage your cluster. @@ -75,6 +86,34 @@ The option can also be specified using the environment variable `K3S_KUBECONFIG_ $ curl -sfL https://get.k3s.io | K3S_KUBECONFIG_MODE="644" sh -s - ``` +### Configuring an Imported EKS Cluster with Terraform + +You should define **only** the minimum fields that Rancher requires when importing an EKS cluster with Terraform. This is important as Rancher will overwrite what was in the EKS cluster with any config that the user has provided. + +>**Warning:** Even a small difference between the current EKS cluster and a user-provided config could have unexpected results. + +The minimum config fields required by Rancher to import EKS clusters with Terraform using `eks_config_v2` are as follows: + +- cloud_credential_id +- name +- region +- imported (this field should always be set to `true` for imported clusters) + +Example YAML configuration for imported EKS clusters: + +``` +resource "rancher2_cluster" "my-eks-to-import" { + name = "my-eks-to-import" + description = "Terraform EKS Cluster" + eks_config_v2 { + cloud_credential_id = rancher2_cloud_credential.aws.id + name = var.aws_eks_name + region = var.aws_region + imported = true + } +} +``` + # Management Capabilities for Registered Clusters The control that Rancher has to manage a registered cluster depends on the type of cluster. @@ -129,7 +168,7 @@ Also in the K3s documentation, nodes with the worker role are called agent nodes # Debug Logging and Troubleshooting for Registered K3s Clusters -Nodes are upgraded by the system upgrade controller running in the downstream cluster. Based on the cluster configuration, Rancher deploys two [plans](https://github.com/rancher/system-upgrade-controller#example-upgrade-plan) to upgrade K3s nodes: one for controlplane nodes and one for workers. The system upgrade controller follows the plans and upgrades the nodes. +Nodes are upgraded by the system upgrade controller running in the downstream cluster. Based on the cluster configuration, Rancher deploys two [plans](https://github.com/rancher/system-upgrade-controller#example-upgrade-plan) to upgrade K3s nodes: one for controlplane nodes and one for workers. The system upgrade controller follows the plans and upgrades the nodes. To enable debug logging on the system upgrade controller deployment, edit the [configmap](https://github.com/rancher/system-upgrade-controller/blob/50a4c8975543d75f1d76a8290001d87dc298bdb4/manifests/system-upgrade-controller.yaml#L32) to set the debug environment variable to true. Then restart the `system-upgrade-controller` pod. @@ -149,8 +188,55 @@ If the cluster becomes stuck in upgrading, restart the `system-upgrade-controlle To prevent issues when upgrading, the [Kubernetes upgrade best practices](https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/) should be followed. +# Authorized Cluster Endpoint Support for RKE2 and K3s Clusters +_Available as of v2.6.3_ +Authorized Cluster Endpoint (ACE) support has been added for registered RKE2 and K3s clusters. This support includes manual steps you will perform on the downstream cluster to enable the ACE. For additional information on the authorized cluster endpoint, click [here]({{}}/rancher/v2.6/en/cluster-admin/cluster-access/ace/). + +> **Note:** +> +> - These steps only need to be performed on the control plane nodes of the downstream cluster. You must configure each control plane node individually. +> +> - The following steps will work on both RKE2 and K3s clusters registered in v2.6.x as well as those registered (or imported) from a previous version of Rancher with an upgrade to v2.6.x. +> +> - These steps will alter the configuration of the downstream RKE2 and K3s clusters and deploy the `kube-api-authn-webhook`. If a future implementation of the ACE requires an update to the `kube-api-authn-webhook`, then this would also have to be done manually. For more information on this webhook, click [here]({{}}/rancher/v2.6/en/cluster-admin/cluster-access/ace/#about-the-kube-api-auth-authentication-webhook). + +###### **Manual steps to be taken on the control plane of each downstream cluster to enable ACE:** + +1. Create a file at `/var/lib/rancher/{rke2,k3s}/kube-api-authn-webhook.yaml` with the following contents: + + apiVersion: v1 + kind: Config + clusters: + - name: Default + cluster: + insecure-skip-tls-verify: true + server: http://127.0.0.1:6440/v1/authenticate + users: + - name: Default + user: + insecure-skip-tls-verify: true + current-context: webhook + contexts: + - name: webhook + context: + user: Default + cluster: Default + +1. Add the following to the config file (or create one if it doesn’t exist); note that the default location is `/etc/rancher/{rke2,k3s}/config.yaml`: + + kube-apiserver-arg: + - authentication-token-webhook-config-file=/var/lib/rancher/{rke2,k3s}/kube-api-authn-webhook.yaml + +1. Run the following commands: + + sudo systemctl stop {rke2,k3s}-server + sudo systemctl start {rke2,k3s}-server + +1. Finally, you **must** go back to the Rancher UI and edit the imported cluster there to complete the ACE enablement. Click on **⋮ > Edit Config**, then click the **Networking** tab under Cluster Configuration. Finally, click the **Enabled** button for **Authorized Endpoint**. Once the ACE is enabled, you then have the option of entering a fully qualified domain name (FQDN) and certificate information. + + >**Note:** The FQDN field is optional, and if one is entered, it should point to the downstream cluster. Certificate information is only needed if there is a load balancer in front of the downstream cluster that is using an untrusted certificate. If you have a valid certificate, then nothing needs to be added to the CA Certificates field. # Annotating Registered Clusters @@ -200,4 +286,3 @@ To annotate a registered cluster, 1. Click **Save**. **Result:** The annotation does not give the capabilities to the cluster, but it does indicate to Rancher that the cluster has those capabilities. - diff --git a/content/rancher/v2.6/en/cluster-provisioning/rke-clusters/_index.md b/content/rancher/v2.6/en/cluster-provisioning/rke-clusters/_index.md index 66fea5d3088..21f9d9de5d5 100644 --- a/content/rancher/v2.6/en/cluster-provisioning/rke-clusters/_index.md +++ b/content/rancher/v2.6/en/cluster-provisioning/rke-clusters/_index.md @@ -3,7 +3,7 @@ title: Launching Kubernetes with Rancher weight: 4 --- -You can have Rancher launch a Kubernetes cluster using any nodes you want. When Rancher deploys Kubernetes onto these nodes, it uses [Rancher Kubernetes Engine]({{}}/rke/latest/en/) (RKE), which is Rancher's own lightweight Kubernetes installer. It can launch Kubernetes on any computers, including: +You can have Rancher launch a Kubernetes cluster using any nodes you want. When Rancher deploys Kubernetes onto these nodes, you can choose between [Rancher Kubernetes Engine]({{}}/rke/latest/en/) (RKE) or [RKE2](https://docs.rke2.io) distributions. Rancher can launch Kubernetes on any computers, including: - Bare-metal servers - On-premise virtual machines @@ -11,29 +11,9 @@ You can have Rancher launch a Kubernetes cluster using any nodes you want. When Rancher can install Kubernetes on existing nodes, or it can dynamically provision nodes in an infrastructure provider and install Kubernetes on them. -RKE clusters include clusters that Rancher launched on Windows nodes or other existing custom nodes, as well as clusters that Rancher launched with new nodes on Azure, Digital Ocean, EC2, or vSphere. +Rancher can also create pools of nodes. One benefit of installing Kubernetes on node pools hosted by an infrastructure provider is that if a node loses connectivity with the cluster, Rancher can automatically create another node to join the cluster to ensure that the count of the node pool is as expected. -### Changes in Rancher v2.6 - -_Tech Preview_ - -Rancher v2.6 introduces provisioning for [RKE2](https://docs.rke2.io/) clusters directly from the Rancher UI. RKE2, also known as RKE Government, is a fully conformant Kubernetes distribution that focuses on security and compliance within the U.S. Federal Government sector. - -RKE2 provisioning is built on top of a new provisioning framework that leverages the upstream [Cluster API](https://github.com/kubernetes-sigs/cluster-api) project. With this new provisioning framework, you can: - -- Provision RKE2 clusters on Digital Ocean, AWS EC2, Azure, and vSphere -- Fully configure RKE2 clusters within Rancher -- Choose CNI options Calico, Cilium, and Multus in addition to Canal -- Install custom RKE2 clusters on pre-provisioned VMs or bare-metal nodes - -The RKE2 provisioning tech preview also includes installing RKE2 on Windows clusters. Windows features for RKE2 include: - -- Windows Containers with RKE2 powered by containerd -- Added provisioning of Windows RKE2 custom clusters directly from the Rancher UI -- Calico CNI for Windows RKE2 custom clusters. -- SAC releases of Windows Server (2004 and 20H2) are included in the technical preview. - -Windows Support for RKE2 Custom Clusters requires choosing Calico as the CNI. +## RKE ### Requirements @@ -54,3 +34,48 @@ In this scenario, you want to install Kubernetes on bare-metal servers, on-prem If you want to reuse a node from a previous custom cluster, [clean the node]({{}}/rancher/v2.6/en/cluster-admin/cleaning-cluster-nodes/) before using it in a cluster again. If you reuse a node that hasn't been cleaned, cluster provisioning may fail. For more information, refer to the section on [custom nodes.]({{}}/rancher/v2.6/en/cluster-provisioning/rke-clusters/custom-nodes/) + +# Programmatically Creating RKE Clusters + +The most common way to programmatically deploy RKE clusters through Rancher is by using the Rancher2 Terraform provider. The documentation for creating clusters with Terraform is [here.](https://registry.terraform.io/providers/rancher/rancher2/latest/docs/resources/cluster) + +## RKE2 + +Rancher v2.6 introduced provisioning for [RKE2](https://docs.rke2.io/) clusters directly from the Rancher UI. RKE2, also known as RKE Government, is a fully conformant Kubernetes distribution that focuses on security and compliance within the U.S. Federal Government sector. In Rancher v.2.6.4 and earlier, RKE2 provisioning was in tech preview. + +Note that in Rancher v2.6.5, RKE2 provisioning became GA. + +### Requirements + +If you use RKE2 to set up a cluster, your nodes must meet the [requirements](https://docs.rke2.io/install/requirements/) for nodes in downstream user clusters. + +### Launching Kubernetes on New Nodes in an Infrastructure Provider + +RKE2 provisioning is built on top of a new provisioning framework that leverages the upstream [Cluster API](https://github.com/kubernetes-sigs/cluster-api) project. With this new provisioning framework, you can: + +- Provision RKE2 clusters onto any provider for which Rancher has a node driver +- Fully configure RKE2 clusters within Rancher +- Choose CNI options Calico, Cilium, and Multus in addition to Canal + +RKE2 provisioning also includes installing RKE2 on clusters with Windows nodes. + +Windows features for RKE2 include: + +- Windows supports the vSphere node driver +- Calico CNI for Windows RKE2 custom clusters +- Project Network Isolation (PNI) for Calico +- Windows Containers with RKE2 powered by containerd +- Provisioning of Windows RKE2 clusters through Terraform +- Provisioning of Windows RKE2 custom clusters directly from the Rancher UI + +Windows Support for RKE2 Custom Clusters requires choosing Calico as the CNI. + +### Launching Kubernetes on Existing Custom Nodes + +RKE2 provisioning also allows you to install custom clusters on pre-provisioned VMs or bare-metal nodes. + +If you want to reuse a node from a previous custom cluster, clean the node before using it in a cluster again. If you reuse a node that hasn't been cleaned, cluster provisioning may fail. + +# Programmatically Creating RKE2 Clusters + +The most common way to programmatically deploy RKE2 clusters through Rancher is by using the Rancher2 Terraform provider. The documentation for creating clusters with Terraform is [here.](https://registry.terraform.io/providers/rancher/rancher2/latest/docs/resources/cluster_v2) \ No newline at end of file diff --git a/content/rancher/v2.6/en/cluster-provisioning/rke-clusters/behavior-differences-between-rke1-and-rke2/_index.md b/content/rancher/v2.6/en/cluster-provisioning/rke-clusters/behavior-differences-between-rke1-and-rke2/_index.md new file mode 100644 index 00000000000..99a3f18f370 --- /dev/null +++ b/content/rancher/v2.6/en/cluster-provisioning/rke-clusters/behavior-differences-between-rke1-and-rke2/_index.md @@ -0,0 +1,34 @@ +--- +title: Behavior Differences Between RKE1 and RKE2 +weight: 2450 +--- + +RKE2, also known as RKE Government, is a Kubernetes distribution that focuses on security and compliance for U.S. Federal Government entities. It is considered the next iteration of the Rancher Kubernetes Engine, now known as RKE1. + +RKE1 and RKE2 have several slight behavioral differences to note, and this page will highlight some of these at a high level. + +### Control Plane Components + +RKE1 uses Docker for deploying and managing control plane components, and it also uses Docker as the container runtime for Kubernetes. By contrast, RKE2 launches control plane components as static pods that are managed by the kubelet. RKE2's container runtime is containerd, which allows things such as container registry mirroring (RKE1 with Docker does not). + +### Cluster API + +RKE2/K3s provisioning is built on top of the Cluster API (CAPI) upstream framework which often makes RKE2-provisioned clusters behave differently than RKE1-provisioned clusters. + +When you make changes to your cluster configuration in RKE2, this **may** result in nodes reprovisioning. This is controlled by CAPI controllers and not by Rancher itself. Note that for etcd nodes, the same behavior does not apply. + +The following are some specific example configuration changes that may cause the described behavior: + +- When editing the cluster and enabling `drain before delete`, the existing control plane nodes and worker are deleted and new nodes are created. + +- When nodes are being provisioned and a scale down operation is performed, rather than scaling down the desired number of nodes, it is possible that the currently provisioning nodes get deleted and new nodes are provisioned to reach the desired node count. Please note that this is a bug in Cluster API, and it will be fixed in an upcoming release. Once fixed, Rancher will update the documentation. + +Users who are used to RKE1 provisioning should take note of this new RKE2 behavior which may be unexpected. + +### Terminology + +You will notice that some terms have changed or gone away going from RKE1 to RKE2. For example, in RKE1 provisioning, you use **node templates**; in RKE2 provisioning, you can configure your cluster node pools when creating or editing the cluster. Another example is that the term **node pool** in RKE1 is now known as **machine pool** in RKE2. + + + + diff --git a/content/rancher/v2.6/en/cluster-provisioning/rke-clusters/cloud-providers/_index.md b/content/rancher/v2.6/en/cluster-provisioning/rke-clusters/cloud-providers/_index.md index 0d82a9a6e0b..bcb6f1d2089 100644 --- a/content/rancher/v2.6/en/cluster-provisioning/rke-clusters/cloud-providers/_index.md +++ b/content/rancher/v2.6/en/cluster-provisioning/rke-clusters/cloud-providers/_index.md @@ -2,7 +2,7 @@ title: Setting up Cloud Providers weight: 2300 --- -A _cloud provider_ is a module in Kubernetes that provides an interface for managing nodes, load balancers, and networking routes. For more information, refer to the [official Kubernetes documentation on cloud providers.](https://kubernetes.io/docs/concepts/cluster-administration/cloud-providers/) +A _cloud provider_ is a module in Kubernetes that provides an interface for managing nodes, load balancers, and networking routes. When a cloud provider is set up in Rancher, the Rancher server can automatically provision new nodes, load balancers or persistent storage devices when launching Kubernetes definitions, if the cloud provider you're using supports such automation. @@ -35,9 +35,9 @@ For details on enabling the vSphere cloud provider, refer to [this page.](./vsph ### Setting up a Custom Cloud Provider -The `Custom` cloud provider is available if you want to configure any [Kubernetes cloud provider](https://kubernetes.io/docs/concepts/cluster-administration/cloud-providers/). +The `Custom` cloud provider is available if you want to configure any Kubernetes cloud provider. -For the custom cloud provider option, you can refer to the [RKE docs]({{}}/rke/latest/en/config-options/cloud-providers/) on how to edit the yaml file for your specific cloud provider. There are specific cloud providers that have more detailed configuration : +For the custom cloud provider option, you can refer to the [RKE docs]({{}}/rke/latest/en/config-options/cloud-providers/) on how to edit the yaml file for your specific cloud provider. There are specific cloud providers that have more detailed configuration: * [vSphere]({{}}/rke/latest/en/config-options/cloud-providers/vsphere/) * [OpenStack]({{}}/rke/latest/en/config-options/cloud-providers/openstack/) diff --git a/content/rancher/v2.6/en/cluster-provisioning/rke-clusters/cloud-providers/vsphere/out-of-tree/_index.md b/content/rancher/v2.6/en/cluster-provisioning/rke-clusters/cloud-providers/vsphere/out-of-tree/_index.md index b9ba43c3c29..d9b95fc9b6f 100644 --- a/content/rancher/v2.6/en/cluster-provisioning/rke-clusters/cloud-providers/vsphere/out-of-tree/_index.md +++ b/content/rancher/v2.6/en/cluster-provisioning/rke-clusters/cloud-providers/vsphere/out-of-tree/_index.md @@ -10,7 +10,10 @@ This page covers how to install the Cloud Provider Interface (CPI) and Cloud Sto # Prerequisites -The vSphere version must be 7.0u1 or higher. +The vSphere versions supported: + +* 6.7u3 +* 7.0u1 or higher. The Kubernetes version must be 1.19 or higher. diff --git a/content/rancher/v2.6/en/cluster-provisioning/rke-clusters/node-pools/_index.md b/content/rancher/v2.6/en/cluster-provisioning/rke-clusters/node-pools/_index.md index 3c89deb26c9..1de5cadd416 100644 --- a/content/rancher/v2.6/en/cluster-provisioning/rke-clusters/node-pools/_index.md +++ b/content/rancher/v2.6/en/cluster-provisioning/rke-clusters/node-pools/_index.md @@ -3,58 +3,55 @@ title: Launching Kubernetes on New Nodes in an Infrastructure Provider weight: 2205 --- +This section covers the following topics: + +- [RKE Clusters](#rke-clusters) + - [Node templates](#node-templates) + - [Node labels](#node-labels) + - [Node taints](#node-taints) + - [Administrator control of node templates](#administrator-control-of-node-templates) + - [Node pools](#node-pools) + - [Node pool taints](#node-pool-taints) + - [About node auto-replace](#about-node-auto-replace) + - [Enabling node auto-replace](#enabling-node-auto-replace) + - [Disabling node auto-replace](#disabling-node-auto-replace) + - [Cloud credentials](#cloud-credentials) + - [Node drivers](#node-drivers) +- [RKE2 Clusters](#rke2-clusters) + - [Node roles in RKE2](#node-roles-in-rke2) + +When you create an RKE or RKE2 cluster using a node template in Rancher, each resulting node pool is shown in a new **Machine Pools** tab. You can see the machine pools by doing the following: + +1. Click **☰ > Cluster Management**. +1. Click the name of the RKE or RKE2 cluster. + +## RKE Clusters + Using Rancher, you can create pools of nodes based on a [node template]({{}}/rancher/v2.6/en/cluster-provisioning/rke-clusters/node-pools/#node-templates). This node template defines the parameters you want to use to launch nodes in your infrastructure providers or cloud providers. One benefit of installing Kubernetes on node pools hosted by an infrastructure provider is that if a node loses connectivity with the cluster, Rancher can automatically create another node to join the cluster to ensure that the count of the node pool is as expected. The available cloud providers to create a node template are decided based on active [node drivers]({{}}/rancher/v2.6/en/cluster-provisioning/rke-clusters/node-pools/#node-drivers). -This section covers the following topics: - -- [Changes in Rancher v2.6](#changes-in-rancher-v2-6) -- [Node templates](#node-templates) - - [Node labels](#node-labels) - - [Node taints](#node-taints) - - [Administrator control of node templates](#administrator-control-of-node-templates) -- [Node pools](#node-pools) - - [Node pool taints](#node-pool-taints) - - [About node auto-replace](#about-node-auto-replace) - - [Enabling node auto-replace](#enabling-node-auto-replace) - - [Disabling node auto-replace](#disabling-node-auto-replace) -- [Cloud credentials](#cloud-credentials) -- [Node drivers](#node-drivers) -- [Node roles in RKE2](#node-roles-in-rke2) - -# Changes in Rancher v2.6 - -_Tech Preview_ - -Rancher v2.6 introduces provisioning for [RKE2](https://docs.rke2.io/) clusters directly from the Rancher UI. RKE2, also known as RKE Government, is a fully conformant Kubernetes distribution that focuses on security and compliance within the U.S. Federal Government sector. - -When you create an RKE or RKE2 cluster using a node template in Rancher, each resulting node pool is shown in a new **Machine Pools** tab. You can see the machine pools by doing the following: - -1. Click **☰ > Cluster Management**. -1. Click the name of the RKE or RKE2 cluster. - -# Node Templates +### Node Templates A node template is the saved configuration for the parameters to use when provisioning nodes in a specific cloud provider. These nodes can be launched from the UI. Rancher uses [Docker Machine](https://docs.docker.com/machine/) to provision these nodes. The available cloud providers to create node templates are based on the active node drivers in Rancher. After you create a node template in Rancher, it's saved so that you can use this template again to create node pools. Node templates are bound to your login. After you add a template, you can remove them from your user profile. -### Node Labels +#### Node Labels You can add [labels](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/) on each node template, so that any nodes created from the node template will automatically have these labels on them. Invalid labels can prevent upgrades or can prevent Rancher from starting. For details on label syntax requirements, see the [Kubernetes documentation.](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#syntax-and-character-set) -### Node Taints +#### Node Taints You can add [taints](https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/) on each node template, so that any nodes created from the node template will automatically have these taints on them. Since taints can be added at a node template and node pool, if there is no conflict with the same key and effect of the taints, all taints will be added to the nodes. If there are taints with the same key and different effect, the taints from the node pool will override the taints from the node template. -### Administrator Control of Node Templates +#### Administrator Control of Node Templates Administrators can control all node templates. Admins can now maintain all the node templates within Rancher. When a node template owner is no longer using Rancher, the node templates created by them can be managed by administrators so the cluster can continue to be updated and maintained. @@ -65,7 +62,7 @@ To access all node templates, an administrator will need to do the following: **Result:** All node templates are listed. The templates can be edited or cloned by clicking the **⋮**. -# Node Pools +### Node Pools Using Rancher, you can create pools of nodes based on a [node template](#node-templates). @@ -75,15 +72,15 @@ The benefit of using a node pool is that if a node is destroyed or deleted, you Each node pool must have one or more nodes roles assigned. -Each node role (i.e. etcd, control plane, and worker) should be assigned to a distinct node pool. Although it is possible to assign multiple node roles to a node pool, this should not be done for production clusters. +Each node role (i.e. etcd, controlplane, and worker) should be assigned to a distinct node pool. Although it is possible to assign multiple node roles to a node pool, this should not be done for production clusters. The recommended setup is to have: - a node pool with the etcd node role and a count of three -- a node pool with the control plane node role and a count of at least two +- a node pool with the controlplane node role and a count of at least two - a node pool with the worker node role and a count of at least two -### Node Pool Taints +#### Node Pool Taints If you haven't defined [taints](https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/) on your node template, you can add taints for each node pool. The benefit of adding taints at a node pool is beneficial over adding it at a node template is that you can swap out the node templates without worrying if the taint is on the node template. @@ -91,7 +88,7 @@ For each taint, they will automatically be added to any created node in the node When there are taints on the node pool and node template, if there is no conflict with the same key and effect of the taints, all taints will be added to the nodes. If there are taints with the same key and different effect, the taints from the node pool will override the taints from the node template. -### About Node Auto-replace +#### About Node Auto-replace If a node is in a node pool, Rancher can automatically replace unreachable nodes. Rancher will use the existing node template for the given node pool to recreate the node if it becomes inactive for a specified number of minutes. @@ -99,7 +96,7 @@ If a node is in a node pool, Rancher can automatically replace unreachable nodes Node auto-replace works on top of the Kubernetes node controller. The node controller periodically checks the status of all the nodes (configurable via the `--node-monitor-period` flag of the `kube-controller`). When a node is unreachable, the node controller will taint that node. When this occurs, Rancher will begin its deletion countdown. You can configure the amount of time Rancher waits to delete the node. If the taint is not removed before the deletion countdown ends, Rancher will proceed to delete the node object. Rancher will then provision a node in accordance with the set quantity of the node pool. -### Enabling Node Auto-replace +#### Enabling Node Auto-replace When you create the node pool, you can specify the amount of time in minutes that Rancher will wait to replace an unresponsive node. @@ -109,7 +106,7 @@ When you create the node pool, you can specify the amount of time in minutes tha **Result:** Node auto-replace is enabled for the node pool. -### Disabling Node Auto-replace +#### Disabling Node Auto-replace You can disable node auto-replace from the Rancher UI with the following steps: @@ -120,7 +117,7 @@ You can disable node auto-replace from the Rancher UI with the following steps: **Result:** Node auto-replace is disabled for the node pool. -# Cloud Credentials +### Cloud Credentials Node templates can use cloud credentials to store credentials for launching nodes in your cloud provider, which has some benefits: @@ -132,20 +129,26 @@ Node templates can use cloud credentials to store credentials for launching node After cloud credentials are created, the user can start [managing the cloud credentials that they created]({{}}/rancher/v2.6/en/user-settings/cloud-credentials/). -# Node Drivers +### Node Drivers If you don't find the node driver that you want to use, you can see if it is available in Rancher's built-in [node drivers and activate it]({{}}/rancher/v2.6/en/admin-settings/drivers/node-drivers/#activating-deactivating-node-drivers), or you can [add your own custom node driver]({{}}/rancher/v2.6/en/admin-settings/drivers/node-drivers/#adding-custom-node-drivers). -# Node Roles in RKE2 +## RKE2 Clusters -The RKE2 CLI exposes two roles, `server` and `agent`, which represent the Kubernetes node-roles `etcd` + `control-plane` and `worker` respectively. With RKE2 integration in Rancher v2.6, RKE2 node pools can represent more fine-grained role assignments such that `etcd` and `control-plane` roles can be represented. +Rancher v2.6 introduces provisioning for [RKE2](https://docs.rke2.io/) clusters directly from the Rancher UI. RKE2, also known as RKE Government, is a fully conformant Kubernetes distribution that focuses on security and compliance within the U.S. Federal Government sector. + +> **Note:** For RKE2 cluster templates, please refer to [this page]({{}}/rancher/v2.6/en/admin-settings/cluster-templates/#rke2-cluster-template) for additional information. + +### Node Roles + +The RKE2 CLI exposes two roles, `server` and `agent`, which represent the Kubernetes node-roles `etcd` + `controlplane` and `worker` respectively. With RKE2 integration in Rancher v2.6, RKE2 node pools can represent more fine-grained role assignments such that `etcd` and `controlplane` roles can be represented. The same functionality of using `etcd`, `controlplane` and `worker` nodes is possible in the RKE2 CLI by using flags and node tainting to control where workloads and the Kubernetes master were scheduled. The reason those roles were not implemented as first-class roles in the RKE2 CLI is that RKE2 is conceptualized as a set of raw building blocks that are best leveraged through an orchestration system such as Rancher. +The implementation of the three node roles in Rancher means that Rancher managed RKE2 clusters are able to easily leverage all of the same architectural best practices that are recommended for RKE clusters. + In our [recommended cluster architecture]({{}}/rancher/v2.6/en/cluster-provisioning/production/recommended-architecture/), we outline how many nodes of each role clusters should have: - At least three nodes with the role etcd to survive losing one node - At least two nodes with the role controlplane for master component high availability -- At least two nodes with the role worker for workload rescheduling upon node failure - -The implementation of the three node roles in Rancher means that Rancher managed RKE2 clusters are able to easily leverage all of the same architectural best practices that are recommended for RKE clusters. \ No newline at end of file +- At least two nodes with the role worker for workload rescheduling upon node failure \ No newline at end of file diff --git a/content/rancher/v2.6/en/cluster-provisioning/rke-clusters/node-pools/azure/_index.md b/content/rancher/v2.6/en/cluster-provisioning/rke-clusters/node-pools/azure/_index.md index c918601b214..8e7deb9bf62 100644 --- a/content/rancher/v2.6/en/cluster-provisioning/rke-clusters/node-pools/azure/_index.md +++ b/content/rancher/v2.6/en/cluster-provisioning/rke-clusters/node-pools/azure/_index.md @@ -6,7 +6,7 @@ weight: 2220 In this section, you'll learn how to install an [RKE]({{}}/rke/latest/en/) Kubernetes cluster in Azure through Rancher. -First, you will set up your Azure cloud credentials in Rancher. Then you will use your cloud credentials to create a node template, which Rancher will use to provision new nodes in Azure. +First, you will set up your Azure cloud credentials in Rancher. Then you will use your cloud credentials to create a node template, which Rancher will use to provision new nodes in Azure. Then you will create an Azure cluster in Rancher, and when configuring the new cluster, you will define node pools for it. Each node pool will have a Kubernetes role of etcd, controlplane, or worker. Rancher will install Kubernetes on the new nodes, and it will set up each node with the Kubernetes role defined by the node pool. @@ -25,29 +25,31 @@ For more information on configuring Azure node templates, refer to the [Azure no - [Creating an Azure Cluster](#creating-an-azure-cluster) # Preparation in Azure - + Before creating a node template in Rancher using a cloud infrastructure such as Azure, we must configure Rancher to allow the manipulation of resources in an Azure subscription. To do this, we will first create a new Azure **service principal (SP)** in Azure **Active Directory (AD)**, which, in Azure, is an application user who has permission to manage Azure resources. The following is a template `az cli` script that you have to run for creating an service principal, where you have to enter your SP name, role, and scope: - + ``` az ad sp create-for-rbac \ --name="" \ --role="Contributor" \ --scopes="/subscriptions/" ``` - + The creation of this service principal returns three pieces of identification information, *The application ID, also called the client ID*, and *The client secret*. This information will be used when you create a node template for Azure. # Creating an Azure Cluster +{{% tabs %}} +{{% tab "RKE" %}} 1. [Create your cloud credentials](#1-create-your-cloud-credentials) 2. [Create a node template with your cloud credentials](#2-create-a-node-template-with-your-cloud-credentials) 3. [Create a cluster with node pools using the node template](#3-create-a-cluster-with-node-pools-using-the-node-template) - + ### 1. Create your cloud credentials 1. Click **☰ > Cluster Management**. @@ -57,7 +59,7 @@ The creation of this service principal returns three pieces of identification in 1. Enter your Azure credentials. 1. Click **Create**. -**Result:** You have created the cloud credentials that will be used to provision nodes in your cluster. You can reuse these credentials for other node templates, or in other clusters. +**Result:** You have created the cloud credentials that will be used to provision nodes in your cluster. You can reuse these credentials for other node templates, or in other clusters. ### 2. Create a node template with your cloud credentials @@ -82,17 +84,53 @@ Use Rancher to create a Kubernetes cluster in Azure. 1. Use **Member Roles** to configure user authorization for the cluster. Click **Add Member** to add users that can access the cluster. Use the **Role** drop-down to set permissions for each user. 1. Click **Create**. -**Result:** +{{% /tab %}} +{{% tab "RKE2" %}} + +### 1. Create your cloud credentials + +If you already have a set of cloud credentials to use, skip this section. + +1. Click **☰ > Cluster Management**. +1. Click **Cloud Credentials**. +1. Click **Create**. +1. Click **Azure**. +1. Enter your Azure credentials. +1. Click **Create**. + +**Result:** You have created the cloud credentials that will be used to provision nodes in your cluster. You can reuse these credentials for other node templates, or in other clusters. + +### 2. Create your cluster + +Use Rancher to create a Kubernetes cluster in Azure. + +1. Click **☰ > Cluster Management**. +1. On the **Clusters** page, click **Create**. +1. Toggle the switch to **RKE2/K3s**. +1. Click **Azure**. +1. Select a **Cloud Credential**, if more than one exists. Otherwise, it's preselected. +1. Enter a **Cluster Name**. +1. Create a machine pool for each Kubernetes role. Refer to the [best practices]({{}}/rancher/v2.6/en/cluster-provisioning/rke-clusters/node-pools#node-roles-in-rke2) for recommendations on role assignments and counts. + 1. For each machine pool, define the machine configuration. Refer to the [Azure machine configuration reference]({{}}/rancher/v2.6/en/cluster-provisioning/rke-clusters/node-pools/azure/azure-machine-config/) for information on configuration options. +1. Use the **Cluster Configuration** to choose the version of Kubernetes that will be installed, what network provider will be used and if you want to enable project network isolation. For help configuring the cluster, refer to the [RKE2 cluster configuration reference.]({{}}/rancher/v2.6/en/cluster-admin/editing-clusters/rke2-config-reference/) +1. Use **Member Roles** to configure user authorization for the cluster. Click **Add Member** to add users that can access the cluster. Use the **Role** drop-down to set permissions for each user. +1. Click **Create**. + +{{% /tab %}} +{{% /tabs %}} + +**Result:** Your cluster is created and assigned a state of **Provisioning**. Rancher is standing up your cluster. You can access your cluster after its state is updated to **Active**. -**Active** clusters are assigned two Projects: +**Active** clusters are assigned two Projects: - `Default`, containing the `default` namespace - `System`, containing the `cattle-system`, `ingress-nginx`, `kube-public`, and `kube-system` namespaces + ### Optional Next Steps After creating your cluster, you can access it through the Rancher UI. As a best practice, we recommend setting up these alternate ways of accessing your cluster: diff --git a/content/rancher/v2.6/en/cluster-provisioning/rke-clusters/node-pools/azure/azure-machine-config/_index.md b/content/rancher/v2.6/en/cluster-provisioning/rke-clusters/node-pools/azure/azure-machine-config/_index.md new file mode 100644 index 00000000000..4c5304eb1e2 --- /dev/null +++ b/content/rancher/v2.6/en/cluster-provisioning/rke-clusters/node-pools/azure/azure-machine-config/_index.md @@ -0,0 +1,121 @@ +--- +title: Azure Machine Configuration +weight: 2 +--- + +For more information about Azure, refer to the official [Azure documentation.](https://docs.microsoft.com/en-us/azure/?product=featured) + +### Environment + +Microsoft provides multiple [clouds](https://docs.microsoft.com/en-us/cli/azure/cloud?view=azure-cli-latest) for compliance with regional laws, which are available for your use: + +- AzurePublicCloud +- AzureGermanCloud +- AzureChinaCloud +- AzureUSGovernmentCloud + +### Location + +Configure the cluster and node [location](https://docs.microsoft.com/en-us/azure/virtual-machines/regions). + +### Resource Group + +A resource group is a container that holds related resources for an Azure solution. The resource group can include all the resources for the solution, or only those resources that you want to manage as a group. You decide how you want to allocate resources to resource groups based on what makes the most sense for your organization. Generally, add resources that share the same lifecycle to the same resource group so you can easily deploy, update, and delete them as a group. + +Use an existing resource group or enter a resource group name and one will be created for you. + +For information on managing resource groups, see the [Azure documentation.](https://docs.microsoft.com/en-us/azure/azure-resource-manager/management/manage-resource-groups-portal) + +### Availability Set (unmanaged) + +Name or ID of an existing [availability set](https://docs.microsoft.com/en-us/azure/virtual-machines/availability-set-overview) to add the VM to. + +### Image + +The name of the operating system image provided as an ARM resource identifier. Requires using managed disk. + +### VM Size + +Choose a size for each VM in the node pool. For details about each VM size, see [this page.](https://azure.microsoft.com/en-us/pricing/details/virtual-machines/linux/) + +## Advanced Options + +### Fault Domain Count + +Fault domains define the group of virtual machines that share a common power source and network switch. If the availability set has already been created, the fault domain count will be ignored. + +For more information on fault domains, see [refer here](https://docs.microsoft.com/en-us/azure/virtual-machines/availability-set-overview#how-do-availability-sets-work). + +### Update Domain Count + +Update domains indicate groups of virtual machines and underlying physical hardware that can be rebooted at the same time. If the availability set has already been created, the update domain count will be ignored. + +For more information on update domains, see [refer here](https://docs.microsoft.com/en-us/azure/virtual-machines/availability-set-overview#how-do-availability-sets-work). + +### Purchase Plan + +Some VM images in the Azure Marketplace require a plan. If applicable, select a purchase plan, formatted as `publisher:product:plan`, to use with your chosen image. + +### Subnet + +The name of the subnet when creating a new VNet or referencing an existing one. + +Default: `docker-machine` + +### Subnet Prefix + +The subnet IP address prefix to use when creating a new VNet in CIDR format. + +Default: `192.168.0.0/16` + +### Virtual Network + +The [virtual network](https://docs.microsoft.com/en-us/azure/virtual-network/virtual-networks-overview) to use or create if one does not exist. Formatted as `[resourcegroup:]name`. + +### Public IP Options + +#### No Public IP + +Do not allocate a public IP address. + +#### Static Public IP + +Allocate a static public IP address. + +### Use Private IP + +Use a static private IP address. + +### Private IP Address + +Configure a static private IP address to use. + +### Network Security Group + +The [network security group](https://docs.microsoft.com/en-us/azure/virtual-network/network-security-groups-overview) to use. All nodes using this template will use the supplied network security group. If no network security group is provided, a new one will be created for each node. + +### DNS Label + +A unique DNS name label for the public IP address. + +### Storage Type + +The [storage account](https://docs.microsoft.com/en-us/azure/storage/common/storage-account-overview) type to use with your VMs. Options include Standard LRS, Standard ZRS, Standard GRS, Standard RAGRS, and Premium LRS. + +### Use Managed Disks + +[Azure managed disks](https://docs.microsoft.com/en-us/azure/virtual-machines/managed-disks-overview) are block-level storage volumes that are managed by Azure and used with Azure Virtual Machines. Managed disks are designed for 99.999% availability. Managed disks achieve this by providing you with three replicas of your data, allowing for high durability. + +### Managed Disk Size + +The size in GB for the disk for each node. + +### SSH Username + +The username used to create an SSH connection to your nodes. + +### Open Port + +Opens inbound traffic on specified ports. When using an existing Network Security Group, Open Ports are ignored. + +Default: `2379/tcp, 2380/tcp, 6443/tcp, 9796/tcp, 10250/tcp, 10251/tcp, 10252/tcp, 10256/tcp` and `8472/udp, 4789/udp` \ No newline at end of file diff --git a/content/rancher/v2.6/en/cluster-provisioning/rke-clusters/node-pools/digital-ocean/_index.md b/content/rancher/v2.6/en/cluster-provisioning/rke-clusters/node-pools/digital-ocean/_index.md index 4c1903b9f63..7d3e103cd1e 100644 --- a/content/rancher/v2.6/en/cluster-provisioning/rke-clusters/node-pools/digital-ocean/_index.md +++ b/content/rancher/v2.6/en/cluster-provisioning/rke-clusters/node-pools/digital-ocean/_index.md @@ -5,10 +5,12 @@ weight: 2215 --- In this section, you'll learn how to use Rancher to install an [RKE](https://rancher.com/docs/rke/latest/en/) Kubernetes cluster in DigitalOcean. -First, you will set up your DigitalOcean cloud credentials in Rancher. Then you will use your cloud credentials to create a node template, which Rancher will use to provision new nodes in DigitalOcean. +First, you will set up your DigitalOcean cloud credentials in Rancher. Then you will use your cloud credentials to create a node template, which Rancher will use to provision new nodes in DigitalOcean. Then you will create a DigitalOcean cluster in Rancher, and when configuring the new cluster, you will define node pools for it. Each node pool will have a Kubernetes role of etcd, controlplane, or worker. Rancher will install RKE Kubernetes on the new nodes, and it will set up each node with the Kubernetes role defined by the node pool. +{{% tabs %}} +{{% tab "RKE" %}} 1. [Create your cloud credentials](#1-create-your-cloud-credentials) 2. [Create a node template with your cloud credentials](#2-create-a-node-template-with-your-cloud-credentials) @@ -23,7 +25,7 @@ Then you will create a DigitalOcean cluster in Rancher, and when configuring the 1. Enter your Digital Ocean credentials. 1. Click **Create**. -**Result:** You have created the cloud credentials that will be used to provision nodes in your cluster. You can reuse these credentials for other node templates, or in other clusters. +**Result:** You have created the cloud credentials that will be used to provision nodes in your cluster. You can reuse these credentials for other node templates, or in other clusters. ### 2. Create a node template with your cloud credentials @@ -46,13 +48,46 @@ Creating a [node template]({{}}/rancher/v2.6/en/cluster-provisioning/rk 1. Use **Member Roles** to configure user authorization for the cluster. Click **Add Member** to add users that can access the cluster. Use the **Role** drop-down to set permissions for each user. 1. Click **Create**. -**Result:** +{{% /tab %}} +{{% tab "RKE2" %}} + +### 1. Create your cloud credentials + +If you already have a set of cloud credentials to use, skip this section. + +1. Click **☰ > Cluster Management**. +1. Click **Cloud Credentials**. +1. Click **Create**. +1. Click **DigitalOcean**. +1. Enter your Digital Ocean credentials. +1. Click **Create**. + +### 2. Create your cluster + +Use Rancher to create a Kubernetes cluster in DigitalOcean. + +1. Click **☰ > Cluster Management**. +1. On the **Clusters** page, click **Create**. +1. Toggle the switch to **RKE2/K3s**. +1. Click **DigitalOcean**. +1. Select a **Cloud Credential**, if more than one exists. Otherwise, it's preselected. +1. Enter a **Cluster Name**. +1. Create a machine pool for each Kubernetes role. Refer to the [best practices]({{}}/rancher/v2.6/en/cluster-provisioning/rke-clusters/node-pools#node-roles-in-rke2) for recommendations on role assignments and counts. + 1. For each machine pool, define the machine configuration. Refer to the [DigitalOcean machine configuration reference]({{}}/rancher/v2.6/en/cluster-provisioning/rke-clusters/node-pools/digital-ocean/do-machine-config/) for information on configuration options. +1. Use the **Cluster Configuration** to choose the version of Kubernetes that will be installed, what network provider will be used and if you want to enable project network isolation. For help configuring the cluster, refer to the [RKE2 cluster configuration reference.]({{}}/rancher/v2.6/en/cluster-admin/editing-clusters/rke2-config-reference/) +1. Use **Member Roles** to configure user authorization for the cluster. Click **Add Member** to add users that can access the cluster. Use the **Role** drop-down to set permissions for each user. +1. Click **Create**. + +{{% /tab %}} +{{% /tabs %}} + +**Result:** Your cluster is created and assigned a state of **Provisioning**. Rancher is standing up your cluster. You can access your cluster after its state is updated to **Active**. -**Active** clusters are assigned two Projects: +**Active** clusters are assigned two Projects: - `Default`, containing the `default` namespace - `System`, containing the `cattle-system`, `ingress-nginx`, `kube-public`, and `kube-system` namespaces @@ -61,4 +96,4 @@ You can access your cluster after its state is updated to **Active**. After creating your cluster, you can access it through the Rancher UI. As a best practice, we recommend setting up these alternate ways of accessing your cluster: - **Access your cluster with the kubectl CLI:** Follow [these steps]({{}}/rancher/v2.6/en/cluster-admin/cluster-access/kubectl/#accessing-clusters-with-kubectl-on-your-workstation) to access clusters with kubectl on your workstation. In this case, you will be authenticated through the Rancher server’s authentication proxy, then Rancher will connect you to the downstream cluster. This method lets you manage the cluster without the Rancher UI. -- **Access your cluster with the kubectl CLI, using the authorized cluster endpoint:** Follow [these steps]({{}}/rancher/v2.6/en/cluster-admin/cluster-access/kubectl/#authenticating-directly-with-a-downstream-cluster) to access your cluster with kubectl directly, without authenticating through Rancher. We recommend setting up this alternative method to access your cluster so that in case you can’t connect to Rancher, you can still access the cluster. \ No newline at end of file +- **Access your cluster with the kubectl CLI, using the authorized cluster endpoint:** Follow [these steps]({{}}/rancher/v2.6/en/cluster-admin/cluster-access/kubectl/#authenticating-directly-with-a-downstream-cluster) to access your cluster with kubectl directly, without authenticating through Rancher. We recommend setting up this alternative method to access your cluster so that in case you can’t connect to Rancher, you can still access the cluster. diff --git a/content/rancher/v2.6/en/cluster-provisioning/rke-clusters/node-pools/digital-ocean/do-machine-config/_index.md b/content/rancher/v2.6/en/cluster-provisioning/rke-clusters/node-pools/digital-ocean/do-machine-config/_index.md new file mode 100644 index 00000000000..0ae2f8d831a --- /dev/null +++ b/content/rancher/v2.6/en/cluster-provisioning/rke-clusters/node-pools/digital-ocean/do-machine-config/_index.md @@ -0,0 +1,34 @@ +--- +title: DigitalOcean Machine Configuration +weight: 2 +--- + +For more details about DigitalOcean, Droplets, refer to the [official documentation](https://docs.digitalocean.com/products/compute/). + +### Region + +Configure the [region](https://docs.digitalocean.com/products/app-platform/concepts/region/) where Droplets are created. + +### Size + +Configure the [size](https://docs.digitalocean.com/products/droplets/resources/choose-plan/) of Droplets. + +### OS Image + +Configure the operating system [image](https://docs.digitalocean.com/products/images/) Droplets are created from. + +### Monitoring + +Enable the DigitalOcean agent for additional [monitoring](https://docs.digitalocean.com/products/monitoring/). + +### IPv6 + +Enable IPv6 for Droplets. + +### Private Networking + +Enable private networking for Droplets. + +### Droplet Tags + +Apply a tag (label) to a Droplet. Tags may only contain letters, numbers, colons, dashes, and underscores. For example, `my_server`. diff --git a/content/rancher/v2.6/en/cluster-provisioning/rke-clusters/node-pools/ec2/_index.md b/content/rancher/v2.6/en/cluster-provisioning/rke-clusters/node-pools/ec2/_index.md index 4e50330f632..26d8d3c4576 100644 --- a/content/rancher/v2.6/en/cluster-provisioning/rke-clusters/node-pools/ec2/_index.md +++ b/content/rancher/v2.6/en/cluster-provisioning/rke-clusters/node-pools/ec2/_index.md @@ -6,7 +6,7 @@ weight: 2210 --- In this section, you'll learn how to use Rancher to install an [RKE](https://rancher.com/docs/rke/latest/en/) Kubernetes cluster in Amazon EC2. -First, you will set up your EC2 cloud credentials in Rancher. Then you will use your cloud credentials to create a node template, which Rancher will use to provision new nodes in EC2. +First, you will set up your EC2 cloud credentials in Rancher. Then you will use your cloud credentials to create a node template, which Rancher will use to provision new nodes in EC2. Then you will create an EC2 cluster in Rancher, and when configuring the new cluster, you will define node pools for it. Each node pool will have a Kubernetes role of etcd, controlplane, or worker. Rancher will install RKE Kubernetes on the new nodes, and it will set up each node with the Kubernetes role defined by the node pool. @@ -23,6 +23,9 @@ Then you will create an EC2 cluster in Rancher, and when configuring the new clu The steps to create a cluster differ based on your Rancher version. +{{% tabs %}} +{{% tab "RKE" %}} + 1. [Create your cloud credentials](#1-create-your-cloud-credentials) 2. [Create a node template with your cloud credentials and information from EC2](#2-create-a-node-template-with-your-cloud-credentials-and-information-from-ec2) 3. [Create a cluster with node pools using the node template](#3-create-a-cluster-with-node-pools-using-the-node-template) @@ -38,7 +41,7 @@ The steps to create a cluster differ based on your Rancher version. 1. Enter your AWS EC2 **Access Key** and **Secret Key**. 1. Click **Create**. -**Result:** You have created the cloud credentials that will be used to provision nodes in your cluster. You can reuse these credentials for other node templates, or in other clusters. +**Result:** You have created the cloud credentials that will be used to provision nodes in your cluster. You can reuse these credentials for other node templates, or in other clusters. ### 2. Create a node template with your cloud credentials and information from EC2 @@ -59,20 +62,55 @@ Add one or more node pools to your cluster. For more information about node pool 1. Click **☰ > Cluster Management**. 1. On the **Clusters** page, click **Create**. 1. Click **Amazon EC2**. -1. Create a node pool for each Kubernetes role. For each node pool, choose a node template that you created. For more information about node pools, including best practices for assigning Kubernetes roles to them, see [this section.]({{}}/rancher/v2.6/en/cluster-provisioning/rke-clusters/node-pools) +1. Create a node pool for each Kubernetes role. For each node pool, choose a node template that you created. For more information about node pools, including best practices for assigning Kubernetes roles to them, see [this section.]({{}}/rancher/v2.6/en/cluster-provisioning/rke-clusters/node-pools) 1. Click **Add Member** to add users that can access the cluster. Use the **Role** drop-down to set permissions for each user. 1. Use **Cluster Options** to choose the version of Kubernetes that will be installed, what network provider will be used and if you want to enable project network isolation. Refer to [Selecting Cloud Providers]({{}}/rancher/v2.6/en/cluster-provisioning/rke-clusters/cloud-providers/) to configure the Kubernetes Cloud Provider. For help configuring the cluster, refer to the [RKE cluster configuration reference.]({{}}/rancher/v2.6/en/cluster-provisioning/rke-clusters/options) >**Note:** If you want to use the [dual-stack](https://kubernetes.io/docs/concepts/services-networking/dual-stack/) feature, there are additional [requirements]({{}}/rke//latest/en/config-options/dual-stack#requirements) that must be taken into consideration. 1. Click **Create**. -**Result:** +{{% /tab %}} +{{% tab "RKE2" %}} + +### 1. Create your cloud credentials + +If you already have a set of cloud credentials to use, skip this section. + +1. Click **☰ > Cluster Management**. +1. Click **Cloud Credentials**. +1. Click **Create**. +1. Click **Amazon**. +1. Enter a name for the cloud credential. +1. In the **Default Region** field, select the AWS region where your cluster nodes will be located. +1. Enter your AWS EC2 **Access Key** and **Secret Key**. +1. Click **Create**. + +**Result:** You have created the cloud credentials that will be used to provision nodes in your cluster. You can reuse these credentials for other node templates, or in other clusters. + +### 2. Create your cluster + +1. Click **☰ > Cluster Management**. +1. On the **Clusters** page, click **Create**. +1. Toggle the switch to **RKE2/K3s**. +1. Click **Amazon EC2**. +1. Select a **Cloud Credential**, if more than one exists. Otherwise, it's preselected. +1. Enter a **Cluster Name**. +1. Create a machine pool for each Kubernetes role. Refer to the [best practices]({{}}/rancher/v2.6/en/cluster-provisioning/rke-clusters/node-pools#node-roles-in-rke2) for recommendations on role assignments and counts. + 1. For each machine pool, define the machine configuration. Refer to [the EC2 machine configuration reference]({{}}/rancher/v2.6/en/cluster-provisioning/rke-clusters/node-pools/ec2/ec2-machine-config/) for information on configuration options. +1. Use the **Cluster Configuration** to choose the version of Kubernetes that will be installed, what network provider will be used and if you want to enable project network isolation. For help configuring the cluster, refer to the [RKE2 cluster configuration reference.]({{}}/rancher/v2.6/en/cluster-admin/editing-clusters/rke2-config-reference/) +1. Use **Member Roles** to configure user authorization for the cluster. Click **Add Member** to add users that can access the cluster. Use the **Role** drop-down to set permissions for each user. +1. Click **Create**. + +{{% /tab %}} +{{% /tabs %}} + +**Result:** Your cluster is created and assigned a state of **Provisioning**. Rancher is standing up your cluster. You can access your cluster after its state is updated to **Active**. -**Active** clusters are assigned two Projects: +**Active** clusters are assigned two Projects: - `Default`, containing the `default` namespace - `System`, containing the `cattle-system`, `ingress-nginx`, `kube-public`, and `kube-system` namespaces diff --git a/content/rancher/v2.6/en/cluster-provisioning/rke-clusters/node-pools/ec2/ec2-machine-config/_index.md b/content/rancher/v2.6/en/cluster-provisioning/rke-clusters/node-pools/ec2/ec2-machine-config/_index.md new file mode 100644 index 00000000000..d29a7567669 --- /dev/null +++ b/content/rancher/v2.6/en/cluster-provisioning/rke-clusters/node-pools/ec2/ec2-machine-config/_index.md @@ -0,0 +1,79 @@ +--- +title: EC2 Machine Configuration Reference +weight: 2 +--- + +For more details about EC2 nodes, refer to the official documentation for the [EC2 Management Console](https://aws.amazon.com/ec2). + +### Region + +The geographical [region](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html) in which to build your cluster. + +### Zone + +The [zone](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html#concepts-availability-zones), an isolated location within a region to build your cluster + +### Instance Type + +The [instance type](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-types.html), which determines the hardware characteristics, used to provision your cluster. + +### Root Disk Size + +Configure the size (in GB) for your [root device](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/RootDeviceStorage.html). + +### VPC/Subnet + +The [VPC](https://docs.aws.amazon.com/vpc/latest/userguide/configure-your-vpc.html) or specific [subnet](https://docs.aws.amazon.com/vpc/latest/userguide/configure-subnets.html), an IP range in your VPC, to add your resources to. + +### IAM Instance Profile Name + +The name of the [instance profile] used to pass an IAM role to an EC2 instance. + +## Advanced Options + +### AMI ID + +The [Amazon Machine Image](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIs.html) used for the nodes in your cluster. + +### SSH Username for AMI + +The username for connecting to your launched instances. Refer to [here](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/connection-prereqs.html) for the default usernames to selected AMIs. For AMIs not listed, check with the AMI provider. + +### Security Group + +Choose the default security group or configure a security group. + +Please refer to [Amazon EC2 security group when using Node Driver]({{}}/rancher/v2.6/en/installation/requirements/ports/#rancher-aws-ec2-security-group) to see what rules are created in the `rancher-nodes` Security Group. + +### EBS Root Volume Type + +The [EBS volume type](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-volume-types.html) to use for the root device. + +### Encrypt EBS Volume + +Enable [Amazon EBS Encryption](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSEncryption.html). + +### Request Spot Instance + +Enable option to [request spot instances](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-requests.html) and specify the maximum instance price per hour you're willing to pay. + +### Use only private address + +Enable option on use only [private addresses](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-instance-addressing.html). + +### EBS-Optimized Instance + +Use an [EBS-optimized instance](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-optimized.html). + +### Allow access to EC2 metadata + +Enable access to [EC2 metadata](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html). + +### Use tokens for metadata + +Use [Instance Metadata Service Version 2 (IMDSv2)](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/configuring-instance-metadata-service.html), a token-based method to access metadata. + +### Add Tag + +Add metadata using [tags](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_Tags.html) to categorize resources. + diff --git a/content/rancher/v2.6/en/cluster-provisioning/rke-clusters/node-pools/ec2/ec2-node-template-config/_index.md b/content/rancher/v2.6/en/cluster-provisioning/rke-clusters/node-pools/ec2/ec2-node-template-config/_index.md index 6e3a1b0dc8c..4da61dccaf7 100644 --- a/content/rancher/v2.6/en/cluster-provisioning/rke-clusters/node-pools/ec2/ec2-node-template-config/_index.md +++ b/content/rancher/v2.6/en/cluster-provisioning/rke-clusters/node-pools/ec2/ec2-node-template-config/_index.md @@ -34,6 +34,11 @@ Choose the default security group or configure a security group. Please refer to [Amazon EC2 security group when using Node Driver]({{}}/rancher/v2.6/en/installation/requirements/ports/#rancher-aws-ec2-security-group) to see what rules are created in the `rancher-nodes` Security Group. +--- +**_New in v2.6.4_** + +If you provide your own security group for an EC2 instance, please note that Rancher will not modify it. As such, you will be responsible for ensuring that your security group is set to allow the [necessary ports for Rancher to provision the instance]({{}}/rancher/v2.6/en/installation/requirements/ports/#ports-for-rancher-server-nodes-on-rke). For more information on controlling inbound and outbound traffic to EC2 instances with security groups, refer [here](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_SecurityGroups.html#WorkingWithSecurityGroups). + ### Instance Options Configure the instances that will be created. Make sure you configure the correct **SSH User** for the configured AMI. It is possible that a selected region does not support the default instance type. In this scenario you must select an instance type that does exist, otherwise an error will occur stating the requested configuration is not supported. diff --git a/content/rancher/v2.6/en/cluster-provisioning/rke-clusters/node-pools/nutanix/_index.md b/content/rancher/v2.6/en/cluster-provisioning/rke-clusters/node-pools/nutanix/_index.md new file mode 100644 index 00000000000..f009dc078f8 --- /dev/null +++ b/content/rancher/v2.6/en/cluster-provisioning/rke-clusters/node-pools/nutanix/_index.md @@ -0,0 +1,20 @@ +--- +title: Creating a Nutanix AOS Cluster +shortTitle: Nutanix +description: Use Rancher to create a Nutanix AOS (AHV) cluster. It may consist of groups of VMs with distinct properties which allow for fine-grained control over the sizing of nodes. +metaDescription: Use Rancher to create a Nutanix AOS (AHV) cluster. It may consist of groups of VMs with distinct properties which allow for fine-grained control over the sizing of nodes. +weight: 2225 +--- + +[Nutanix Acropolis Operating System](https://www.nutanix.com/products/acropolis) (Nutanix AOS) is an operating system for the Nutanix hyper-converged infrastructure platform. AOS comes with a built-in hypervisor called [Acropolis Hypervisor](https://www.nutanix.com/products/ahv), or AHV. By using Rancher with Nutanix AOS (AHV), you can bring cloud operations on-premises. + +Rancher can provision nodes in AOS (AHV) and install Kubernetes on them. When creating a Kubernetes cluster in AOS, Rancher first provisions the specified number of virtual machines by communicating with the Prism Central API. Then it installs Kubernetes on top of the VMs. + +A Nutanix cluster may consist of multiple groups of VMs with distinct properties, such as the amount of memory or the number of vCPUs. This grouping allows for fine-grained control over the sizing of nodes for each Kubernetes role. + +- [Creating a Nutanix Cluster]({{}}/rancher/v2.6/en/cluster-provisioning/rke-clusters/node-pools/nutanix/provisioning-nutanix-clusters/#creating-a-nutanix-aos-cluster) +- [Provisioning Storage]({{}}/rancher/v2.6/en/cluster-provisioning/rke-clusters/node-pools/nutanix/provisioning-nutanix-clusters) + +# Creating a Nutanix Cluster + +In [this section,](./provisioning-nutanix-clusters) you'll learn how to use Rancher to install an [RKE]({{}}/rke/latest/en/) Kubernetes cluster in Nutanix AOS. \ No newline at end of file diff --git a/content/rancher/v2.6/en/cluster-provisioning/rke-clusters/node-pools/nutanix/nutanix-node-template-config/_index.md b/content/rancher/v2.6/en/cluster-provisioning/rke-clusters/node-pools/nutanix/nutanix-node-template-config/_index.md new file mode 100644 index 00000000000..a4ae10d8ece --- /dev/null +++ b/content/rancher/v2.6/en/cluster-provisioning/rke-clusters/node-pools/nutanix/nutanix-node-template-config/_index.md @@ -0,0 +1,64 @@ +--- +title: Nutanix Node Template Configuration +weight: 2 +--- + +- [Account Access](#account-access) +- [Scheduling](#scheduling) +- [Instance Options](#instance-options) +- [Networks](#networks) +- [VM Categories](#vm-categories) +- [cloud-init](#cloud-init) + +# Account Access + +| Parameter | Required | Description | Default +|:-----------------------------|:--------:|:-----------------------------------------------------------------|:----- +| Management Endpoint | ✓ | Hostname/IP address of Prism Central | +| Username | ✓ | Username of the Prism Central user | +| Password | ✓ | Password of the Prism Central user | +| Allow insecure communication | | Set to true to allow insecure SSL communication to Prism Central | False + +# Scheduling + +Choose what Nutanix cluster the virtual machine will be scheduled to. + +| Parameter | Required | Description +|:----------|:--------:|:---------------------------------------------------------------------------- +| Cluster | ✓ | Name of the Nutanix cluster where the VM should be deployed (case sensitive) + +# Instance Options + +In the **Instance Options** section, configure the number of vCPUs, memory, and disk size for the VMs created by this template. + +| Parameter | Required | Description | Default +|:---------------------|:--------:|:--------------------------------------------------------------------------------------------|:------- +| CPUs | | Number of vCPUs allocated to the VM (cores) | 2 +| Memory | | Amount of RAM allocated to the VM (MB) | 2 GB +| Template Image | ✓ | Name of the Disk Image template to clone as the VM's primary disk (must support cloud-init) | +| VM Disk Size | | New size of the VM's primary disk (in GiB) | +| Additional Disk Size | | Size of an additional disk to add to the VM (in GiB) | +| Storage Container | | Storage container _UUID_ in which to provision an additional disk | +| Cloud Config YAML | | Cloud-init to provide to the VM (will be patched with Rancher root user) | +| Network | ✓ | Name(s) of the network(s) to attach to the VM | +| VM Categories | | Name(s) of any categories to be applied to the VM | + +The VM may use any modern Linux operating system that is configured with support for [cloud-init](https://cloudinit.readthedocs.io/en/latest/) using the [Config Drive v2 datasource](https://cloudinit.readthedocs.io/en/latest/topics/datasources/configdrive.html). + +# Networks + +The node template allows a VM to be provisioned with multiple networks. In the **Network** field, you can click **Add** to add any networks available to you in AOS. + +# VM Categories + +A category is a grouping of entities into a key value pair. Typically, VMs are assigned to a category based on some criteria. Policies can then be tied to those entities that are assigned (grouped by) a specific category value. + +# cloud-init + +[Cloud-init](https://cloudinit.readthedocs.io/en/latest/) allows you to initialize your nodes by applying configuration on the first boot. This may involve things such as creating users or authorizing SSH keys. + +To make use of cloud-init initialization, paste a cloud config using valid YAML syntax into the **Cloud Config YAML** field. Refer to the [cloud-init documentation](https://cloudinit.readthedocs.io/en/latest/topics/examples.html) for a commented set of examples of supported cloud config directives. + +Note that cloud-init based network configuration is not recommended and only supported via user data `runcmd` rather than by NoCloud or other network configuration datasources. + +Nutanix IP Address Management (IPAM) or another DHCP service is recommended. diff --git a/content/rancher/v2.6/en/cluster-provisioning/rke-clusters/node-pools/nutanix/provisioning-nutanix-clusters/_index.md b/content/rancher/v2.6/en/cluster-provisioning/rke-clusters/node-pools/nutanix/provisioning-nutanix-clusters/_index.md new file mode 100644 index 00000000000..de2647ddced --- /dev/null +++ b/content/rancher/v2.6/en/cluster-provisioning/rke-clusters/node-pools/nutanix/provisioning-nutanix-clusters/_index.md @@ -0,0 +1,90 @@ +--- +title: Provisioning Kubernetes Clusters in Nutanix AOS +weight: 1 +--- + +To use Rancher to install an [RKE]({{}}/rke/latest/en/) Kubernetes cluster in Nutanix AOS (AHV): + +1. Locate Rancher's built-in Nutanix [node driver and activate it]({{}}/rancher/v2.6/en/admin-settings/drivers/node-drivers/#activating-deactivating-node-drivers). + +1. Create a node template, which Rancher will use to provision nodes in Nutanix AOS. + +1. Create a Nutanix AOS cluster in Rancher. When configuring the new cluster, you will define node pools for it. Each node pool will have a Kubernetes role of etcd, controlplane, or worker. Rancher will install RKE Kubernetes on the new nodes, and it will set up each node with the Kubernetes role defined by the node pool. + +For details on configuring the Nutanix AOS node template, refer to the [Nutanix AOS node template configuration reference.]({{}}/rancher/v2.6/en/cluster-provisioning/rke-clusters/node-pools/nutanix/nutanix-node-template-config/) + +For details on configuring RKE Kubernetes clusters in Rancher, refer to the [cluster configuration reference.]({{}}/rancher/v2.6/en/cluster-admin/editing-clusters/rke-config-reference/) + +- [Preparation in Nutanix AOS](#preparation-in-nutanix-aos) +- [Creating a Nutanix AOS Cluster](#creating-a-nutanix-aos-cluster) + +# Preparation in Nutanix AOS + +The following sections describe the requirements for setting up Nutanix AOS so that Rancher can provision VMs and clusters. + +**Note:** The node templates are documented and tested with Nutanix AOS version 5.20.2 and 6.0.1. + +### Create Credentials in Nutanix AOS + +Before proceeding to create a cluster, you must ensure that you have a [Nutanix Prism Central user account](https://portal.nutanix.com/page/documents/details?targetId=Nutanix-Security-Guide-v6_0:wc-user-create-wc-t.html) with admin permissions. When you set up a node template, the template will need to use these credentials. + +### Network Permissions + +You must ensure that the hosts running the Rancher server are able to establish the following network connections: + +- To the Nutanix Prism Central API (usually port 9440/TCP). +- To port 22/TCP and 2376/TCP on the created VMs + +See [Node Networking Requirements]({{}}/rancher/v2.6/en/cluster-provisioning/node-requirements/#networking-requirements) for a detailed list of port requirements applicable for creating nodes on an infrastructure provider. + +### VM-VM Anti-Affinity Policies + +Setting up [VM-VM Anti-Affinity Policies](https://portal.nutanix.com/page/documents/details?targetId=AHV-Admin-Guide-v6_1:ahv-vm-anti-affinity-t.html) is recommended. These rules allow VMs assigned the etcd and control-plane roles to operate on separate AHV hosts when they are assigned to different node pools. This practice ensures that the failure of a single physical machine does not affect the availability of those planes. + +# Creating a Nutanix AOS Cluster + +1. [Create a node template ](#1-create-a-node-template) +2. [Create a cluster with node pools using the node template](#2-create-a-cluster-with-node-pools-using-the-node-template) + +### 1. Create a node template + +Creating a [node template]({{}}/rancher/v2.6/en/cluster-provisioning/rke-clusters/node-pools/#node-templates) for Nutanix AOS will allow Rancher to provision new nodes in Nutanix AOS. Node templates can be reused for other clusters. + +1. Click **☰ > Cluster Management**. +1. Click **RKE1 Configuration > Node Templates**. +1. Click **Create**. +1. Click **Add Template**. +1. Click **Nutanix**. +1. Fill out a node template for Nutanix AOS. For help filling out the form, refer to the Nutanix AOS node template [configuration reference.]({{}}/rancher/v2.6/en/cluster-provisioning/rke-clusters/node-pools/nutanix/nutanix-node-template-config/). +1. Click **Create**. + +### 2. Create a cluster with node pools using the node template + +Use Rancher to create a Kubernetes cluster in Nutanix AOS. + +1. Click **☰ > Cluster Management**. +1. On the **Clusters** page, click **Create**. +1. Click **Nutanix**. +1. Enter a **Cluster Name**, then click **Continue**. +1. Use **Member Roles** to configure user authorization for the cluster. Click **Add Member** to add users who can access the cluster. Use the **Role** drop-down to set permissions for each user. +1. Use **Cluster Options** to choose the version of Kubernetes that will be installed, what network provider will be used, and whether you want to enable project network isolation. To see more cluster options, click on **Show advanced options**. For help configuring the cluster, refer to the [RKE cluster configuration reference.]({{}}/rancher/v2.6/en/cluster-admin/editing-clusters/rke-config-reference/) +1. Add one or more node pools to your cluster. Each node pool uses a node template to provision new nodes. For more information about node pools, including best practices for assigning Kubernetes roles to the nodes, see [this section.]({{}}/rancher/v2.6/en/cluster-provisioning/rke-clusters/node-pools/#node-pools) +1. Review your options to confirm they're correct. Then click **Create**. + +**Result:** Your cluster is created and assigned a state of **Provisioning**. Rancher is standing up your cluster. + +You can access your cluster after its state is updated to **Active**. + +**Active** clusters are assigned two Projects: + +- `Default`, containing the `default` namespace +- `System`, containing the `cattle-system`, `ingress-nginx`, `kube-public`, and `kube-system` namespaces + + +# Optional Next Steps + +After creating your cluster, you can access it through the Rancher UI. As a best practice, we recommend setting up these alternate ways of accessing your cluster: + +- **Access your cluster with the kubectl CLI:** Follow [these steps]({{}}/rancher/v2.6/en/cluster-admin/cluster-access/kubectl/#accessing-clusters-with-kubectl-on-your-workstation) to access clusters with kubectl on your workstation. In this case, you will be authenticated through the Rancher server’s authentication proxy, then Rancher will connect you to the downstream cluster. This method lets you manage the cluster without the Rancher UI. + +- **Access your cluster with the kubectl CLI, using the authorized cluster endpoint:** Follow [these steps]({{}}/rancher/v2.6/en/cluster-admin/cluster-access/kubectl/#authenticating-directly-with-a-downstream-cluster) to access your cluster with kubectl directly, without authenticating through Rancher. We recommend setting up this alternative method to access your cluster so that in case you can’t connect to Rancher, you can still access the cluster. \ No newline at end of file diff --git a/content/rancher/v2.6/en/cluster-provisioning/rke-clusters/node-pools/vsphere/_index.md b/content/rancher/v2.6/en/cluster-provisioning/rke-clusters/node-pools/vsphere/_index.md index 13c897c89b3..e11d83949fb 100644 --- a/content/rancher/v2.6/en/cluster-provisioning/rke-clusters/node-pools/vsphere/_index.md +++ b/content/rancher/v2.6/en/cluster-provisioning/rke-clusters/node-pools/vsphere/_index.md @@ -36,6 +36,7 @@ For the fields to be populated, your setup needs to fulfill the [prerequisites.] ### More Supported Operating Systems You can provision VMs with any operating system that supports `cloud-init`. Only YAML format is supported for the [cloud config.](https://cloudinit.readthedocs.io/en/latest/topics/examples.html) + ### Video Walkthrough of v2.3.3 Node Template Features In this YouTube video, we demonstrate how to set up a node template with the new features designed to help you bring cloud operations to on-premises clusters. @@ -54,4 +55,4 @@ For an example of how to provision storage in vSphere using Rancher, refer to [t When a cloud provider is set up in Rancher, the Rancher server can automatically provision new infrastructure for the cluster, including new nodes or persistent storage devices. -For details, refer to the section on [enabling the vSphere cloud provider.]({{}}/rancher/v2.6/en/cluster-provisioning/rke-clusters/cloud-providers/vsphere) \ No newline at end of file +For details, refer to the section on [enabling the vSphere cloud provider.]({{}}/rancher/v2.6/en/cluster-provisioning/rke-clusters/cloud-providers/vsphere) diff --git a/content/rancher/v2.6/en/cluster-provisioning/rke-clusters/node-pools/vsphere/creating-a-vm-template/_index.md b/content/rancher/v2.6/en/cluster-provisioning/rke-clusters/node-pools/vsphere/creating-a-vm-template/_index.md new file mode 100644 index 00000000000..1ed401c2ebd --- /dev/null +++ b/content/rancher/v2.6/en/cluster-provisioning/rke-clusters/node-pools/vsphere/creating-a-vm-template/_index.md @@ -0,0 +1,160 @@ +--- +title: Creating a vSphere Virtual Machine Template +weight: 4 +--- + +Creating virtual machines in a repeatable and reliable fashion can often be difficult. VMware vSphere offers the ability to build one VM that can then be converted to a template. The template can then be used to create identically configured VMs. Rancher leverages this capability within node pools to create identical RKE1 and RKE2 nodes. + +In order to leverage the template to create new VMs, Rancher has some [specific requirements](#requirements) that the VM must have pre-installed. After you configure the VM with these requirements, you will next need to [prepare the VM](#preparing-your-vm) before [creating the template](#creating-a-template). Finally, once preparation is complete, the VM can be [converted to a template](#converting-to-a-template) and [moved into a content library](#moving-to-a-content-library), ready for Rancher node pool usage. + +- [Requirements](#requirements) +- [Creating a Template](#creating-a-template) +- [Preparing Your VM](#preparing-your-vm) +- [Converting to a Template](#converting-to-a-template) +- [Moving to a content library](#moving-to-a-content-library) +- [Other Resources](#other-resources) + +# Requirements + +There is specific tooling required for both Linux and Windows VMs to be usable by the vSphere node driver. The most critical dependency is [cloud-init](https://cloud-init.io/) for Linux and [cloudbase-init](https://cloudbase.it/cloudbase-init/) for Windows. Both of these are used for provisioning the VMs by configuring the hostname and by setting up the SSH access and the default Rancher user. Users can add additional content to these as desired if other configuration is needed. In addition, other requirements are listed below for reference. + +**Note:** If you have any specific firewall rules or configuration, you will need to add this to the VM before creating a template. + +## Linux Dependencies + +The packages that need to be installed on the template are listed below. These will have slightly different names based on distribution; some distributions ship these by default, for example. + +* curl +* wget +* git +* net-tools +* unzip +* apparmor-parser +* ca-certificates +* cloud-init +* cloud-guest-utils +* cloud-image-utils +* growpart +* cloud-initramfs-growroot +* open-iscsi +* openssh-server +* [open-vm-tools](https://docs.vmware.com/en/VMware-Tools/11.3.0/com.vmware.vsphere.vmwaretools.doc/GUID-8B6EA5B7-453B-48AA-92E5-DB7F061341D1.html) + +## Windows Dependencies + +The list of packages that need to be installed on the template is as follows: + +* Windows Container Feature +* [cloudbase-init](https://cloudbase.it/cloudbase-init/#download) +* [Docker EE](https://docs.microsoft.com/en-us/virtualization/windowscontainers/quick-start/set-up-environment?tabs=Windows-Server#install-docker) - RKE1 Only + +**Important to note: The configuration for Windows templates varies between RKE1 and RKE2:** +- RKE1 leverages Docker, so any RKE1 templates need to have Docker EE pre-installed as well +- RKE2 does not require Docker EE, and thus it does not need to be installed + +# Creating a Template + +You may either manually create your VM or you can utilize [other alternatives](#alternatives-to-manual-creation) to create your VM. + +## Manual Creation +1. Manually create your VM by following [these instructions](https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vm_admin.doc/GUID-AE8AFBF1-75D1-4172-988C-378C35C9FAF2.html) from VMware. Once you have a VM running, you can manually install the dependencies listed above to configure the VM correctly for the vSphere node driver. +2. Customize as needed based on your specific environment and requirements. +3. Proceed with the final preparation before creating your template. + +## Alternatives to Manual Creation + +Other alternative options to create VMs are listed below: + +* [VMware PowerCLI](https://developer.vmware.com/powercli) +* [Packer](https://www.packer.io/) +* [SaltStack](https://saltproject.io/) +* [Ansible](https://www.ansible.com/) + +Packer is a frequently-used alternative. Refer to this [reference](https://github.com/vmware-samples/packer-examples-for-vsphere) for examples of its usage with vSphere. + +# Preparing Your VM + +After creating a VM with all the required dependencies (and any additional required items), you must perform the most critical step next: preparing the VM to be turned into a template. This preparation will reset critical data such as the VM hostname, IPs, etc., to prevent that information from being brought into a new VM. If you fail to perform this step, you could create a VM with the same hostname, IP address, etc. + +Note that these preparatory steps differ between Linux and Windows. + +## Linux Preparation + +The commands below will reset your VM in Linux: + +```Bash +# Cleaning logs. +if [ -f /var/log/audit/audit.log ]; then + cat /dev/null > /var/log/audit/audit.log +fi +if [ -f /var/log/wtmp ]; then + cat /dev/null > /var/log/wtmp +fi +if [ -f /var/log/lastlog ]; then + cat /dev/null > /var/log/lastlog +fi + +# Cleaning udev rules. +if [ -f /etc/udev/rules.d/70-persistent-net.rules ]; then + rm /etc/udev/rules.d/70-persistent-net.rules +fi + +# Cleaning the /tmp directories +rm -rf /tmp/* +rm -rf /var/tmp/* + +# Cleaning the SSH host keys +rm -f /etc/ssh/ssh_host_* + +# Cleaning the machine-id +truncate -s 0 /etc/machine-id +rm /var/lib/dbus/machine-id +ln -s /etc/machine-id /var/lib/dbus/machine-id + +# Cleaning the shell history +unset HISTFILE +history -cw +echo > ~/.bash_history +rm -fr /root/.bash_history + +# Truncating hostname, hosts, resolv.conf and setting hostname to localhost +truncate -s 0 /etc/{hostname,hosts,resolv.conf} +hostnamectl set-hostname localhost + +# Clean cloud-init +cloud-init clean -s -l +``` + +## Windows Preparation + +Windows has a utility called [sysprep](https://docs.microsoft.com/en-us/windows-hardware/manufacture/desktop/sysprep--generalize--a-windows-installation) that is used to generalize an image and reset the same items listed above for Linux. The command is as follows: + +```PowerShell +sysprep.exe /generalize /shutdown /oobe +``` + +# Converting to a Template + +1. Shut down and stop the VM. +2. Right-click on the VM in the inventory list and select **Template**. +3. Click on **Convert to Template**. + +**Result:** Once the process has completed, a template will be available for use. + +For additional information on converting a VM to a template, see the [VMware guide](https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vm_admin.doc/GUID-5B3737CC-28DB-4334-BD18-6E12011CDC9F.html). + +# Moving to a Content library + +Rancher has the ability to use templates provided by a content library. Content libraries store and manage content within vSphere, and they also offer the ability to publish and share that content. + +Below are some helpful links on content libraries: + +* [Create a content library](https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vm_admin.doc/GUID-2A0F1C13-7336-45CE-B211-610D39A6E1F4.html) +* [Clone the template to the content library](https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vm_admin.doc/GUID-AC1545F0-F8BA-4CD2-96EB-21B3DFAA1DC1.html) + +# Other Resources + +Here is a list of additional resources that may be useful: + +* [Tutorial for creating a Linux template](https://docs.microsoft.com/en-us/azure/cloud-adoption-framework/manage/hybrid/server/best-practices/vmware-ubuntu-template) +* [Tutorial for creating a Windows template](https://docs.microsoft.com/en-us/azure/cloud-adoption-framework/manage/hybrid/server/best-practices/vmware-windows-template) diff --git a/content/rancher/v2.6/en/cluster-provisioning/rke-clusters/node-pools/vsphere/creating-credentials/_index.md b/content/rancher/v2.6/en/cluster-provisioning/rke-clusters/node-pools/vsphere/creating-credentials/_index.md index 6f83c84c66c..55ac548274e 100644 --- a/content/rancher/v2.6/en/cluster-provisioning/rke-clusters/node-pools/vsphere/creating-credentials/_index.md +++ b/content/rancher/v2.6/en/cluster-provisioning/rke-clusters/node-pools/vsphere/creating-credentials/_index.md @@ -10,9 +10,11 @@ The following table lists the permissions required for the vSphere user account: | Privilege Group | Operations | |:----------------------|:-----------------------------------------------------------------------| | Datastore | AllocateSpace
Browse
FileManagement (Low level file operations)
UpdateVirtualMachineFiles
UpdateVirtualMachineMetadata | +| Global | Set custom attribute | | Network | Assign | | Resource | AssignVMToPool | | Virtual Machine | Config (All)
GuestOperations (All)
Interact (All)
Inventory (All)
Provisioning (All) | +| vSphere Tagging | Assign or Unassign vSphere Tag
Assign or Unassign vSphere Tag on Object | The following steps create a role with the required privileges and then assign it to a new user in the vSphere console: diff --git a/content/rancher/v2.6/en/cluster-provisioning/rke-clusters/windows-clusters/_index.md b/content/rancher/v2.6/en/cluster-provisioning/rke-clusters/windows-clusters/_index.md index 1d5b9d4b072..5db4add3d41 100644 --- a/content/rancher/v2.6/en/cluster-provisioning/rke-clusters/windows-clusters/_index.md +++ b/content/rancher/v2.6/en/cluster-provisioning/rke-clusters/windows-clusters/_index.md @@ -28,13 +28,13 @@ This guide covers the following topics: - [Configuration for Storage Classes in Azure](#configuration-for-storage-classes-in-azure) - # Changes in Rancher v2.6 - -_Tech Preview_ +# Changes in Rancher v2.6 Rancher v2.6 introduces provisioning for [RKE2](https://docs.rke2.io/) clusters directly from the Rancher UI. RKE2, also known as RKE Government, is a fully conformant Kubernetes distribution that focuses on security and compliance within the U.S. Federal Government sector. -The RKE2 provisioning tech preview also includes installing RKE2 on Windows clusters. Windows features for RKE2 include: +As of Rancher v2.6.5, provisioning for RKE2 is GA. + +The RKE2 provisioning feature also includes installing RKE2 on Windows clusters. Windows features for RKE2 include: - Windows Containers with RKE2 powered by containerd - Added provisioning of Windows RKE2 custom clusters directly from the Rancher UI @@ -43,6 +43,8 @@ The RKE2 provisioning tech preview also includes installing RKE2 on Windows clus Windows Support for RKE2 Custom Clusters requires choosing Calico as the CNI. +>**Important:** Rancher will allow Windows workload pods to deploy on both Windows and Linux worker nodes by default. When creating mixed clusters in RKE2, you must edit the `nodeSelector` in the chart to direct the pods to be placed onto a compatible Windows node. Refer to the [Kubernetes documentation](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#nodeselector) for more information on how to use `nodeSelector` to assign pods to nodes. + # Requirements for Windows Clusters The general node requirements for networking, operating systems, and Docker are the same as the node requirements for a [Rancher installation]({{}}/rancher/v2.6/en/installation/requirements/). @@ -85,6 +87,32 @@ If you are configuring DHCP options sets for an AWS virtual private cloud, note > Some Linux operating systems accept multiple domain names separated by spaces. However, other Linux operating systems and Windows treat the value as a single domain, which results in unexpected behavior. If your DHCP options set is associated with a VPC that has instances with multiple operating systems, specify only one domain name. +### Rancher on vSphere with ESXi 6.7u2 and above + +If you are using Rancher on VMware vSphere with ESXi 6.7u2 or later with Red Hat Enterprise Linux 8.3, CentOS 8.3, or SUSE Enterprise Linux 15 SP2 or later, it is necessary to disable the `vmxnet3` virtual network adapter hardware offloading feature. Failure to do so will result in all network connections between pods on different cluster nodes to fail with timeout errors. All connections from Windows pods to critical services running on Linux nodes, such as CoreDNS, will fail as well. It is also possible that external connections may fail. This issue is the result of Linux distributions enabling the hardware offloading feature in `vmxnet3` and a bug in the `vmxnet3` hardware offloading feature that results in the discarding of packets for guest overlay traffic. To address this issue, it is necessary disable the `vmxnet3` hardware offloading feature. This setting does not survive reboot, so it is necessary to disable on every boot. The recommended course of action is to create a systemd unit file at `/etc/systemd/system/disable_hw_offloading.service`, which disables the `vmxnet3` hardware offloading feature on boot. A sample systemd unit file which disables the `vmxnet3` hardware offloading feature is as follows. Note that `` must be customized to the host `vmxnet3` network interface, e.g., `ens192`: + +``` +[Unit] +Description=Disable vmxnet3 hardware offloading feature + +[Service] +Type=oneshot +ExecStart=ethtool -K tx-udp_tnl-segmentation off +ExecStart=ethtool -K tx-udp_tnl-csum-segmentation off +StandardOutput=journal + +[Install] +WantedBy=multi-user.target +``` +Then set the appropriate permissions on the systemd unit file: +``` +chmod 0644 /etc/systemd/system/disable_hw_offloading.service +``` +Finally, enable the systemd service: +``` +systemctl enable disable_hw_offloading.service +``` + ### Architecture Requirements The Kubernetes cluster management nodes (`etcd` and `controlplane`) must be run on Linux nodes. diff --git a/content/rancher/v2.6/en/cluster-provisioning/rke-clusters/windows-clusters/windows-parity/_index.md b/content/rancher/v2.6/en/cluster-provisioning/rke-clusters/windows-clusters/windows-parity/_index.md index 1a6e9f9f215..658b48e8b18 100644 --- a/content/rancher/v2.6/en/cluster-provisioning/rke-clusters/windows-clusters/windows-parity/_index.md +++ b/content/rancher/v2.6/en/cluster-provisioning/rke-clusters/windows-clusters/windows-parity/_index.md @@ -11,7 +11,7 @@ The following chart describes the feature parity between Windows and Linux on Ra --- | --- | --- **Distributions** | | RKE | Supported | Supported -RKE2 | Supported | Tenatively Planned For 2.6.x +RKE2 | Supported | Supported K3S | Supported | Not Supported EKS | Supported | Not Supported GKE | Supported | Not Supported @@ -38,7 +38,7 @@ Backup/Restore Operator | Supported | Not Supported **CNI / Add-ons** | | Flannel | Supported | Supported Canal | Supported | Not Supported -Calico | Supported | Tentatively Planned for 2.6.x +Calico | Supported | Supported (RKE2 Only) Cilium | Supported | Not Supported Multus | Supported | Not Supported Traefik | Supported | Not Supported diff --git a/content/rancher/v2.6/en/contributing/_index.md b/content/rancher/v2.6/en/contributing/_index.md index 3900cabb57f..cb8d8332c8e 100644 --- a/content/rancher/v2.6/en/contributing/_index.md +++ b/content/rancher/v2.6/en/contributing/_index.md @@ -24,7 +24,7 @@ Repository | URL | Description Rancher | https://github.com/rancher/rancher | This repository is the main source code for Rancher 2.x. Types | https://github.com/rancher/types | This repository is the repository that has all the API types for Rancher 2.x. API Framework | https://github.com/rancher/norman | This repository is an API framework for building Rancher style APIs backed by Kubernetes Custom Resources. -User Interface | https://github.com/rancher/ui | This repository is the source of the UI. +User Interface | https://github.com/rancher/dashboard/ | This repository is the source of the Dashboard UI. (Rancher) Docker Machine | https://github.com/rancher/machine | This repository is the source of the Docker Machine binary used when using Node Drivers. This is a fork of the `docker/machine` repository. machine-package | https://github.com/rancher/machine-package | This repository is used to build the Rancher Docker Machine binary. kontainer-engine | https://github.com/rancher/kontainer-engine | This repository is the source of kontainer-engine, the tool to provision hosted Kubernetes clusters. @@ -36,7 +36,7 @@ loglevel repository | https://github.com/rancher/loglevel | This repository is t To see all libraries/projects used in Rancher, see the [`go.mod` file](https://github.com/rancher/rancher/blob/master/go.mod) in the `rancher/rancher` repository. -![Rancher diagram]({{}}/img/rancher/ranchercomponentsdiagram.svg)
+![Rancher diagram]({{}}/img/rancher/ranchercomponentsdiagram-2.6.svg)
Rancher components used for provisioning/managing Kubernetes clusters. # Building diff --git a/content/rancher/v2.6/en/deploy-across-clusters/fleet/_index.md b/content/rancher/v2.6/en/deploy-across-clusters/fleet/_index.md index f9ef3cc818c..1b129ee97f0 100644 --- a/content/rancher/v2.6/en/deploy-across-clusters/fleet/_index.md +++ b/content/rancher/v2.6/en/deploy-across-clusters/fleet/_index.md @@ -12,6 +12,8 @@ Fleet is a separate project from Rancher, and can be installed on any Kubernetes - [Windows Support](#windows-support) - [GitHub Repository](#github-repository) - [Using Fleet Behind a Proxy](#using-fleet-behind-a-proxy) +- [Helm Chart Dependencies](#helm-chart-dependencies) +- [Troubleshooting](#troubleshooting) - [Documentation](#documentation) # Architecture @@ -20,7 +22,24 @@ For information about how Fleet works, see [this page.](./architecture) # Accessing Fleet in the Rancher UI -Fleet comes preinstalled in Rancher. To access it, click **☰ > Continuous Delivery**. +Fleet comes preinstalled in Rancher and is managed by the **Continous Delivery** option in the Rancher UI. For additional information on Continuous Delivery and other Fleet troubleshooting tips, refer [here](https://fleet.rancher.io/troubleshooting/). + +Users can leverage continuous delivery to deploy their applications to the Kubernetes clusters in the git repository without any manual operation by following **gitops** practice. + +Follow the steps below to access Continuous Delivery in the Rancher UI: + +1. Click **☰ > Continuous Delivery**. + +1. Select your namespace at the top of the menu, noting the following: + - By default,`fleet-default` is selected which includes all downstream clusters that are registered through Rancher. + - You may switch to `fleet-local`, which only contains the `local` cluster, or you may create your own workspace to which you may assign and move clusters. + - You can then manage clusters by clicking on **Clusters** on the left navigation bar. + +1. Click on **Gitrepos** on the left navigation bar to deploy the gitrepo into your clusters in the current workspace. + +1. Select your [git repository](https://fleet.rancher.io/gitrepo-add/) and [target clusters/cluster group](https://fleet.rancher.io/gitrepo-structure/). You can also create the cluster group in the UI by clicking on **Cluster Groups** from the left navigation bar. + +1. Once the gitrepo is deployed, you can monitor the application through the Rancher UI. # Windows Support @@ -36,6 +55,22 @@ The Fleet Helm charts are available [here.](https://github.com/rancher/fleet/rel For details on using Fleet behind a proxy, see [this page.](./proxy) +# Helm Chart Dependencies + +In order for Helm charts with dependencies to deploy successfully, you must run a manual command (as listed below), as it is up to the user to fulfill the dependency list. If you do not do this and proceed to clone your repository and run `helm install`, your installation will fail because the dependencies will be missing. + +The Helm chart in the git repository must include its dependencies in the charts subdirectory. You must either manually run `helm dependencies update $chart` OR run `helm dependencies build $chart` locally, then commit the complete charts directory to your git repository. Note that you will update your commands with the applicable parameters. + +# Troubleshooting + +--- +* **Known Issue:** clientSecretName and helmSecretName secrets for Fleet gitrepos are not included in the backup nor restore created by the [backup-restore-operator]({{}}/rancher/v2.6/en/backups/back-up-rancher/#1-install-the-rancher-backups-operator). We will update the community once a permanent solution is in place. + +* **Temporary Workaround:**
+By default, user-defined secrets are not backed up in Fleet. It is necessary to recreate secrets if performing a disaster recovery restore or migration of Rancher into a fresh cluster. To modify resourceSet to include extra resources you want to backup, refer to docs [here](https://github.com/rancher/backup-restore-operator#user-flow). + +--- + # Documentation The Fleet documentation is at [https://fleet.rancher.io/.](https://fleet.rancher.io/) diff --git a/content/rancher/v2.6/en/faq/dockershim/_index.md b/content/rancher/v2.6/en/faq/dockershim/_index.md new file mode 100644 index 00000000000..a9f79c0bddd --- /dev/null +++ b/content/rancher/v2.6/en/faq/dockershim/_index.md @@ -0,0 +1,46 @@ +--- +title: Dockershim +weight: 300 +--- + +The Dockershim is the CRI compliant layer between the Kubelet and the Docker daemon. As part of the Kubernetes 1.20 release, the [deprecation of the in-tree Dockershim was announced](https://kubernetes.io/blog/2020/12/02/dont-panic-kubernetes-and-docker/). Removal is currently scheduled for Kubernetes 1.24. For more information on the deprecation and its timelines, see the [Kubernetes Dockershim Deprecation FAQ](https://kubernetes.io/blog/2020/12/02/dockershim-faq/#when-will-dockershim-be-removed). + +RKE clusters, starting with Kubernetes 1.21, now support the external Dockershim to continue leveraging Docker as the CRI runtime. We now implement the upstream open source community Dockershim announced by [Mirantis and Docker](https://www.mirantis.com/blog/mirantis-to-take-over-support-of-kubernetes-dockershim-2/) to ensure RKE clusters can continue to leverage Docker. + +To enable the external Dockershim, configure the following option. + +``` +enable_cri_dockerd: true +``` + +For users looking to use another container runtime, Rancher has the edge-focused K3s and datacenter-focused RKE2 Kubernetes distributions that use containerd as the default runtime. Imported RKE2 and K3s Kubernetes clusters can then be upgraded and managed through Rancher even after the removal of in-tree Dockershim in Kubernetes 1.24. + +### FAQ + +
+ +Q. Do I have to upgrade Rancher to get Rancher’s support of the upstream Dockershim? + +The upstream support of Dockershim begins for RKE in Kubernetes 1.21. You will need to be on Rancher 2.6 or above to have support for RKE with Kubernetes 1.21. See our [support matrix](https://rancher.com/support-maintenance-terms/all-supported-versions/rancher-v2.6.0/) for details. + +
+ +Q. I am currently on RKE with Kubernetes 1.20. Do I need to upgrade to RKE with Kubernetes 1.21 sooner to avoid being out of support for Dockershim? + +A. The version of Dockershim in RKE with Kubernetes 1.20 will continue to work and is not scheduled for removal upstream until Kubernetes 1.24. It will only emit a warning of its future deprecation, which Rancher has mitigated in RKE with Kubernetes 1.21. You can plan your upgrade to Kubernetes 1.21 as you would normally, but should consider enabling the external Dockershim by Kubernetes 1.22. The external Dockershim will need to be enabled before upgrading to Kubernetes 1.24, at which point the existing implementation will be removed. + +For more information on the deprecation and its timeline, see the [Kubernetes Dockershim Deprecation FAQ](https://kubernetes.io/blog/2020/12/02/dockershim-faq/#when-will-dockershim-be-removed). + +
+ +Q: What are my other options if I don’t want to depend on the Dockershim? + +A: You can use a runtime like containerd with Kubernetes that does not require Dockershim support. RKE2 or K3s are two options for doing this. + +
+ +Q: If I am already using RKE1 and want to switch to RKE2, what are my migration options? + +A: Rancher is exploring the possibility of an in-place upgrade path. Alternatively you can always migrate workloads from one cluster to another using kubectl. + +
diff --git a/content/rancher/v2.6/en/faq/networking/cni-providers/_index.md b/content/rancher/v2.6/en/faq/networking/cni-providers/_index.md index 498e63ad5f1..ea4bbde4872 100644 --- a/content/rancher/v2.6/en/faq/networking/cni-providers/_index.md +++ b/content/rancher/v2.6/en/faq/networking/cni-providers/_index.md @@ -14,11 +14,11 @@ Kubernetes uses CNI as an interface between network providers and Kubernetes pod For more information visit [CNI GitHub project](https://github.com/containernetworking/cni). -### What Network Models are Used in CNI? +## What Network Models are Used in CNI? -CNI network providers implement their network fabric using either an encapsulated network model such as Virtual Extensible Lan ([VXLAN](https://github.com/coreos/flannel/blob/master/Documentation/backends.md#vxlan)) or an unencapsulated network model such as Border Gateway Protocol ([BGP](https://en.wikipedia.org/wiki/Border_Gateway_Protocol)). +CNI network providers implement their network fabric using either an encapsulated network model such as Virtual Extensible Lan ([VXLAN](https://github.com/flannel-io/flannel/blob/master/Documentation/backends.md#vxlan)) or an unencapsulated network model such as Border Gateway Protocol ([BGP](https://en.wikipedia.org/wiki/Border_Gateway_Protocol)). -#### What is an Encapsulated Network? +### What is an Encapsulated Network? This network model provides a logical Layer 2 (L2) network encapsulated over the existing Layer 3 (L3) network topology that spans the Kubernetes cluster nodes. With this model you have an isolated L2 network for containers without needing routing distribution, all at the cost of minimal overhead in terms of processing and increased IP package size, which comes from an IP header generated by overlay encapsulation. Encapsulation information is distributed by UDP ports between Kubernetes workers, interchanging network control plane information about how MAC addresses can be reached. Common encapsulation used in this kind of network model is VXLAN, Internet Protocol Security (IPSec), and IP-in-IP. @@ -26,11 +26,11 @@ In simple terms, this network model generates a kind of network bridge extended This network model is used when an extended L2 bridge is preferred. This network model is sensitive to L3 network latencies of the Kubernetes workers. If datacenters are in distinct geolocations, be sure to have low latencies between them to avoid eventual network segmentation. -CNI network providers using this network model include Flannel, Canal, and Weave. +CNI network providers using this network model include Flannel, Canal, Weave, and Cilium. By default, Calico is not using this model, but it can be configured to do so. ![Encapsulated Network]({{}}/img/rancher/encapsulated-network.png) -#### What is an Unencapsulated Network? +### What is an Unencapsulated Network? This network model provides an L3 network to route packets between containers. This model doesn't generate an isolated l2 network, nor generates overhead. These benefits come at the cost of Kubernetes workers having to manage any route distribution that's needed. Instead of using IP headers for encapsulation, this network model uses a network protocol between Kubernetes workers to distribute routing information to reach pods, such as [BGP](https://en.wikipedia.org/wiki/Border_Gateway_Protocol). @@ -38,13 +38,17 @@ In simple terms, this network model generates a kind of network router extended This network model is used when a routed L3 network is preferred. This mode dynamically updates routes at the OS level for Kubernetes workers. It's less sensitive to latency. -CNI network providers using this network model include Calico and Romana. +CNI network providers using this network model include Calico and Cilium. Cilium may be configured with this model although it is not the default mode. ![Unencapsulated Network]({{}}/img/rancher/unencapsulated-network.png) -### What CNI Providers are Provided by Rancher? +## What CNI Providers are Provided by Rancher? -Out-of-the-box, Rancher provides the following CNI network providers for Kubernetes clusters: Canal, Flannel, Calico and Weave. You can choose your CNI network provider when you create new Kubernetes clusters from Rancher. +### RKE Kubernetes clusters + +Out-of-the-box, Rancher provides the following CNI network providers for RKE Kubernetes clusters: Canal, Flannel, and Weave. + +You can choose your CNI network provider when you create new Kubernetes clusters from Rancher. #### Canal @@ -54,7 +58,7 @@ Canal is a CNI network provider that gives you the best of Flannel and Calico. I In Rancher, Canal is the default CNI network provider combined with Flannel and VXLAN encapsulation. -Kubernetes workers should open UDP port `8472` (VXLAN) and TCP port `9099` (healthcheck). For details, refer to [the port requirements for user clusters.]({{}}/rancher/v2.6/en/cluster-provisioning/node-requirements/) +Kubernetes workers should open UDP port `8472` (VXLAN) and TCP port `9099` (health checks). If using Wireguard, you should open UDP ports `51820` and `51821`. For more details, refer to [the port requirements for user clusters]({{}}/rancher/v2.6/en/cluster-provisioning/node-requirements/). {{< img "/img/rancher/canal-diagram.png" "Canal Diagram">}} @@ -64,33 +68,18 @@ For more information, see the [Canal GitHub Page.](https://github.com/projectcal ![Flannel Logo]({{}}/img/rancher/flannel-logo.png) -Flannel is a simple and easy way to configure L3 network fabric designed for Kubernetes. Flannel runs a single binary agent named flanneld on each host, which is responsible for allocating a subnet lease to each host out of a larger, preconfigured address space. Flannel uses either the Kubernetes API or etcd directly to store the network configuration, the allocated subnets, and any auxiliary data (such as the host's public IP). Packets are forwarded using one of several backend mechanisms, with the default encapsulation being [VXLAN](https://github.com/coreos/flannel/blob/master/Documentation/backends.md#vxlan). +Flannel is a simple and easy way to configure L3 network fabric designed for Kubernetes. Flannel runs a single binary agent named flanneld on each host, which is responsible for allocating a subnet lease to each host out of a larger, preconfigured address space. Flannel uses either the Kubernetes API or etcd directly to store the network configuration, the allocated subnets, and any auxiliary data (such as the host's public IP). Packets are forwarded using one of several backend mechanisms, with the default encapsulation being [VXLAN](https://github.com/flannel-io/flannel/blob/master/Documentation/backends.md#vxlan). -Encapsulated traffic is unencrypted by default. Therefore, flannel provides an experimental backend for encryption, [IPSec](https://github.com/coreos/flannel/blob/master/Documentation/backends.md#ipsec), which makes use of [strongSwan](https://www.strongswan.org/) to establish encrypted IPSec tunnels between Kubernetes workers. +Encapsulated traffic is unencrypted by default. Flannel provides two solutions for encryption: -Kubernetes workers should open UDP port `8472` (VXLAN) and TCP port `9099` (healthcheck). See [the port requirements for user clusters]({{}}/rancher/v2.6/en/cluster-provisioning/node-requirements/#networking-requirements) for more details. +* [IPSec](https://github.com/flannel-io/flannel/blob/master/Documentation/backends.md#ipsec), which makes use of [strongSwan](https://www.strongswan.org/) to establish encrypted IPSec tunnels between Kubernetes workers. It is an experimental backend for encryption. +* [WireGuard](https://github.com/flannel-io/flannel/blob/master/Documentation/backends.md#wireguard), which is a more faster-performing alternative to strongSwan. + +Kubernetes workers should open UDP port `8472` (VXLAN). See [the port requirements for user clusters]({{}}/rancher/v2.6/en/cluster-provisioning/node-requirements/#networking-requirements) for more details. ![Flannel Diagram]({{}}/img/rancher/flannel-diagram.png) -For more information, see the [Flannel GitHub Page](https://github.com/coreos/flannel). - -#### Calico - -![Calico Logo]({{}}/img/rancher/calico-logo.png) - -Calico enables networking and network policy in Kubernetes clusters across the cloud. Calico uses a pure, unencapsulated IP network fabric and policy engine to provide networking for your Kubernetes workloads. Workloads are able to communicate over both cloud infrastructure and on-prem using BGP. - -Calico also provides a stateless IP-in-IP encapsulation mode that can be used, if necessary. Calico also offers policy isolation, allowing you to secure and govern your Kubernetes workloads using advanced ingress and egress policies. - -Kubernetes workers should open TCP port `179` (BGP). See [the port requirements for user clusters]({{}}/rancher/v2.6/en/cluster-provisioning/node-requirements/#networking-requirements) for more details. - -![Calico Diagram]({{}}/img/rancher/calico-diagram.svg) - -For more information, see the following pages: - -- [Project Calico Official Site](https://www.projectcalico.org/) -- [Project Calico GitHub Page](https://github.com/projectcalico/calico) - +For more information, see the [Flannel GitHub Page](https://github.com/flannel-io/flannel). #### Weave @@ -104,16 +93,73 @@ For more information, see the following pages: - [Weave Net Official Site](https://www.weave.works/) -### CNI Features by Provider +### RKE2 Kubernetes clusters + +Out-of-the-box, Rancher provides the following CNI network providers for RKE2 Kubernetes clusters: [Canal](#canal) (see above section), Calico, and Cilium. + +You can choose your CNI network provider when you create new Kubernetes clusters from Rancher. + +#### Calico + +![Calico Logo]({{}}/img/rancher/calico-logo.png) + +Calico enables networking and network policy in Kubernetes clusters across the cloud. By default, Calico uses a pure, unencapsulated IP network fabric and policy engine to provide networking for your Kubernetes workloads. Workloads are able to communicate over both cloud infrastructure and on-prem using BGP. + +Calico also provides a stateless IP-in-IP or VXLAN encapsulation mode that can be used, if necessary. Calico also offers policy isolation, allowing you to secure and govern your Kubernetes workloads using advanced ingress and egress policies. + +Kubernetes workers should open TCP port `179` if using BGP or UDP port `4789` if using VXLAN encapsulation. In addition, TCP port `5473` is needed when using Typha. See [the port requirements for user clusters]({{}}/rancher/v2.6/en/cluster-provisioning/node-requirements/#networking-requirements) for more details. + +>**Important:** In Rancher v2.6.3, Calico probes fail on Windows nodes upon RKE2 installation. Note that this issue is resolved in v2.6.4. +> +>- To work around this issue, first navigate to `https:///v3/settings/windows-rke2-install-script`. +> +>- There, change the current setting: `https://raw.githubusercontent.com/rancher/wins/v0.1.3/install.ps1` to this new setting: `https://raw.githubusercontent.com/rancher/rke2/master/windows/rke2-install.ps1`. + +![Calico Diagram]({{}}/img/rancher/calico-diagram.svg) + +For more information, see the following pages: + +- [Project Calico Official Site](https://www.projectcalico.org/) +- [Project Calico GitHub Page](https://github.com/projectcalico/calico) + +#### Cilium + +![Cilium Logo]({{}}/img/rancher/cilium-logo.png) + +Cilium enables networking and network policies (L3, L4, and L7) in Kubernetes. By default, Cilium uses eBPF technologies to route packets inside the node and VXLAN to send packets to other nodes. Unencapsulated techniques can also be configured. + +Cilium recommends kernel versions greater than 5.2 to be able to leverage the full potential of eBPF. Kubernetes workers should open TCP port `8472` for VXLAN and TCP port `4240` for health checks. In addition, ICMP 8/0 must be enabled for health checks. For more information, check [Cilium System Requirements](https://docs.cilium.io/en/latest/operations/system_requirements/#firewall-requirements). + +##### Ingress Routing Across Nodes in Cilium +
+By default, Cilium does not allow pods to contact pods on other nodes. To work around this, enable the ingress controller to route requests across nodes with a `CiliumNetworkPolicy`. + +After selecting the Cilium CNI and enabling Project Network Isolation for your new cluster, configure as follows: + +``` +apiVersion: cilium.io/v2 +kind: CiliumNetworkPolicy +metadata: + name: hn-nodes + namespace: default +spec: + endpointSelector: {} + ingress: + - fromEntities: + - remote-node +``` + +## CNI Features by Provider The following table summarizes the different features available for each CNI network provider provided by Rancher. | Provider | Network Model | Route Distribution | Network Policies | Mesh | External Datastore | Encryption | Ingress/Egress Policies | | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | -| Canal | Encapsulated (VXLAN) | No | Yes | No | K8S API | No | Yes | -| Flannel | Encapsulated (VXLAN) | No | No | No | K8S API | No | No | -| Calico | Encapsulated (VXLAN,IPIP) OR Unencapsulated | Yes | Yes | Yes | Etcd and K8S API | No | Yes | +| Canal | Encapsulated (VXLAN) | No | Yes | No | K8s API | Yes | Yes | +| Flannel | Encapsulated (VXLAN) | No | No | No | K8s API | Yes | No | +| Calico | Encapsulated (VXLAN,IPIP) OR Unencapsulated | Yes | Yes | Yes | Etcd and K8s API | Yes | Yes | | Weave | Encapsulated | Yes | Yes | Yes | No | Yes | Yes | +| Cilium | Encapsulated (VXLAN) | Yes | Yes | Yes | Etcd and K8s API | Yes | Yes | - Network Model: Encapsulated or unencapsulated. For more information, see [What Network Models are Used in CNI?](#what-network-models-are-used-in-cni) @@ -129,25 +175,27 @@ The following table summarizes the different features available for each CNI net - Ingress/Egress Policies: This feature allows you to manage routing control for both Kubernetes and non-Kubernetes communications. -#### CNI Community Popularity -The following table summarizes different GitHub metrics to give you an idea of each project's popularity and activity. This data was collected in January 2020. +## CNI Community Popularity + +The following table summarizes different GitHub metrics to give you an idea of each project's popularity and activity. This data was collected in January 2022. | Provider | Project | Stars | Forks | Contributors | | ---- | ---- | ---- | ---- | ---- | -| Canal | https://github.com/projectcalico/canal | 614 | 89 | 19 | -| flannel | https://github.com/coreos/flannel | 4977 | 1.4k | 140 | -| Calico | https://github.com/projectcalico/calico | 1534 | 429 | 135 | -| Weave | https://github.com/weaveworks/weave/ | 5737 | 559 | 73 | +| Canal | https://github.com/projectcalico/canal | 679 | 100 | 21 | +| Flannel | https://github.com/flannel-io/flannel | 7k | 2.5k | 185 | +| Calico | https://github.com/projectcalico/calico | 3.1k | 741 | 224 | +| Weave | https://github.com/weaveworks/weave/ | 6.2k | 635 | 84 | +| Cilium | https://github.com/cilium/cilium | 10.6k | 1.3k | 352 |
-### Which CNI Provider Should I Use? +## Which CNI Provider Should I Use? It depends on your project needs. There are many different providers, which each have various features and options. There isn't one provider that meets everyone's needs. Canal is the default CNI network provider. We recommend it for most use cases. It provides encapsulated networking for containers with Flannel, while adding Calico network policies that can provide project/namespace isolation in terms of networking. -### How can I configure a CNI network provider? +## How can I configure a CNI network provider? Please see [Cluster Options]({{}}/rancher/v2.6/en/cluster-provisioning/rke-clusters/options/) on how to configure a network provider for your cluster. For more advanced configuration options, please see how to configure your cluster using a [Config File]({{}}/rancher/v2.6/en/cluster-provisioning/rke-clusters/options/#cluster-config-file) and the options for [Network Plug-ins]({{}}/rke/latest/en/config-options/add-ons/network-plugins/). diff --git a/content/rancher/v2.6/en/helm-charts/_index.md b/content/rancher/v2.6/en/helm-charts/_index.md index f61e0c2ecfe..aab00dd91dd 100644 --- a/content/rancher/v2.6/en/helm-charts/_index.md +++ b/content/rancher/v2.6/en/helm-charts/_index.md @@ -5,6 +5,43 @@ weight: 11 In this section, you'll learn how to manage Helm chart repositories and applications in Rancher. Helm chart repositories are managed using **Apps & Marketplace**. It uses a catalog-like system to import bundles of charts from repositories and then uses those charts to either deploy custom Helm applications or Rancher's tools such as Monitoring or Istio. Rancher tools come as pre-loaded repositories which deploy as standalone Helm charts. Any additional repositories are only added to the current cluster. +### Changes in Rancher v2.6 + +Starting in Rancher v2.6.0, a new versioning scheme for Rancher feature charts was implemented. The changes are centered around the major version of the charts and the +up annotation for upstream charts, where applicable. + +**Major Version:** The major version of the charts is tied to Rancher minor versions. When you upgrade to a new Rancher minor version, you should ensure that all of your **Apps & Marketplace** charts are also upgraded to the correct release line for the chart. + +>**Note:** Any major versions that are less than the ones mentioned in the table below are meant for 2.5 and below only. For example, you are advised to not use <100.x.x versions of Monitoring in 2.6.x+. + +**Feature Charts:** + +| **Name** | **Supported Minimum Version** | **Supported Maximum Version** | +| ---------------- | ------------ | ------------ | +| external-ip-webhook | 100.0.0+up1.0.0 | 100.0.1+up1.0.1 | +| harvester-cloud-provider | 100.0.2+up0.1.12 | 100.0.2+up0.1.12 | +| harvester-csi-driver | 100.0.2+up0.1.11 | 100.0.2+up0.1.11 | +| rancher-alerting-drivers | 100.0.0 | 100.0.2 | +| rancher-backup | 2.0.1 | 2.1.2 | +| rancher-cis-benchmark | 2.0.1 | 2.0.4 | +| rancher-gatekeeper | 100.0.0+up3.6.0 | 100.1.0+up3.7.1 | +| rancher-istio | 100.0.0+up1.10.4 | 100.2.0+up1.12.6 | +| rancher-logging | 100.0.0+up3.12.0 | 100.1.2+up3.17.4 | +| rancher-longhorn | 100.0.0+up1.1.2 | 100.1.1+up1.2.3 | +| rancher-monitoring | 100.0.0+up16.6.0 | 100.1.0+up19.0.3 +| rancher-sriov (experimental) | 100.0.0+up0.1.0 | 100.0.3+up0.1.0 | +| rancher-vsphere-cpi | 100.3.0+up1.2.1 | 100.3.0+up1.2.1 | +| rancher-vsphere-csi | 100.3.0+up2.5.1-rancher1 | 100.3.0+up2.5.1-rancher1 | +| rancher-wins-upgrader | 0.0.100 | 100.0.0+up0.0.1 | +| neuvector | 100.0.0+up2.2.0 | 100.0.0+up2.2.0 | + +
+**Charts based on upstream:** For charts that are based on upstreams, the +up annotation should inform you of what upstream version the Rancher chart is tracking. Check the upstream version compatibility with Rancher during upgrades also. + +- As an example, `100.x.x+up16.6.0` for Monitoring tracks upstream kube-prometheus-stack `16.6.0` with some Rancher patches added to it. + +- On upgrades, ensure that you are not downgrading the version of the chart that you are using. For example, if you are using a version of Monitoring > `16.6.0` in Rancher 2.5, you should not upgrade to `100.x.x+up16.6.0`. Instead, you should upgrade to the appropriate version in the next release. + + ### Charts From the top-left menu select _"Apps & Marketplace"_ and you will be taken to the Charts page. @@ -25,6 +62,44 @@ From the left sidebar select _"Repositories"_. These items represent helm repositories, and can be either traditional helm endpoints which have an index.yaml, or git repositories which will be cloned and can point to a specific branch. In order to use custom charts, simply add your repository here and they will become available in the Charts tab under the name of the repository. +To add a private CA for Helm Chart repositories: + +- **HTTP-based chart repositories**: You must add a base64 encoded copy of the CA certificate in DER format to the spec.caBundle field of the chart repo, such as `openssl x509 -outform der -in ca.pem | base64 -w0`. Click **Edit YAML** for the chart repo and set, as in the following example:
+ ``` + [...] + spec: + caBundle: + MIIFXzCCA0egAwIBAgIUWNy8WrvSkgNzV0zdWRP79j9cVcEwDQYJKoZIhvcNAQELBQAwPzELMAkGA1UEBhMCVVMxCzAJBgNVBAgMAkNBMRQwEgYDVQQKDAtNeU9yZywgSW5jLjENMAsGA1UEAwwEcm9vdDAeFw0yMTEyMTQwODMyMTdaFw0yNDEwMDMwODMyMT + ... + nDxZ/tNXt/WPJr/PgEB3hQdInDWYMg7vGO0Oz00G5kWg0sJ0ZTSoA10ZwdjIdGEeKlj1NlPyAqpQ+uDnmx6DW+zqfYtLnc/g6GuLLVPamraqN+gyU8CHwAWPNjZonFN9Vpg0PIk1I2zuOc4EHifoTAXSpnjfzfyAxCaZsnTptimlPFJJqAMj+FfDArGmr4= + [...] + ``` + + +- **Git-based chart repositories**: You must add a base64 encoded copy of the CA certificate in DER format to the spec.caBundle field of the chart repo, such as `openssl x509 -outform der -in ca.pem | base64 -w0`. Click **Edit YAML** for the chart repo and set, as in the following example:
+ ``` + [...] + spec: + caBundle: + MIIFXzCCA0egAwIBAgIUWNy8WrvSkgNzV0zdWRP79j9cVcEwDQYJKoZIhvcNAQELBQAwPzELMAkGA1UEBhMCVVMxCzAJBgNVBAgMAkNBMRQwEgYDVQQKDAtNeU9yZywgSW5jLjENMAsGA1UEAwwEcm9vdDAeFw0yMTEyMTQwODMyMTdaFw0yNDEwMDMwODMyMT + ... + nDxZ/tNXt/WPJr/PgEB3hQdInDWYMg7vGO0Oz00G5kWg0sJ0ZTSoA10ZwdjIdGEeKlj1NlPyAqpQ+uDnmx6DW+zqfYtLnc/g6GuLLVPamraqN+gyU8CHwAWPNjZonFN9Vpg0PIk1I2zuOc4EHifoTAXSpnjfzfyAxCaZsnTptimlPFJJqAMj+FfDArGmr4= + [...] + ``` + + +> **Note:** Helm chart repositories with authentication +> +> As of Rancher v2.6.3, a new value `disableSameOriginCheck` has been added to the Repo.Spec. This allows users to bypass the same origin checks, sending the repository Authentication information as a Basic Auth Header with all API calls. This is not recommended but can be used as a temporary solution in cases of non-standard Helm chart repositories such as those that have redirects to a different origin URL. +> +> To use this feature for an existing Helm chart repository, click ⋮ > Edit YAML. On the `spec` portion of the YAML file, add `disableSameOriginCheck` and set it to `true`. +> +> ```yaml +[...] +spec: + disableSameOriginCheck: true +[...] +``` ### Helm Compatibility @@ -42,7 +117,21 @@ After installing a chart, you can find it in the _"Installed Apps"_ tab. In this Most Rancher tools have additional pages located in the toolbar below the _"Apps & Marketplace"_ section to help manage and use the features. These pages include links to dashboards, forms to easily add Custom Resources, and additional information. > If you are upgrading your chart using _"Customize Helm options before upgrade"_ , please be aware that using the _"--force"_ option may result in errors if your chart has immutable fields. This is because some objects in Kubernetes cannot be changed once they are created. To ensure you do not get this error you can: -> +> > * use the default upgrade option ( i.e do not use _"--force"_ option ) > * uninstall the existing chart and install the upgraded chart > * delete the resources with immutable fields from the cluster before performing the _"--force"_ upgrade + +#### Changes in Rancher v2.6.3 + +The upgrade button has been removed for legacy apps from the **Apps & Marketplace > Installed Apps** page. + +If you have a legacy app installed and want to upgrade it: + +- The legacy [feature flag]({{}}/rancher/v2.6/en/installation/resources/feature-flags/) must be turned on (if it's not turned on automatically because of having a legacy app before upgrading) +- You can upgrade the app from cluster explorer, from the left nav section **Legacy > Project > Apps** +- For multi-cluster apps, you can go to **≡ > Multi-cluster Apps** and upgrade the app from there + +### Limitations + +[Dashboard apps or Rancher feature charts](../helm-charts/) **cannot** be installed using the Rancher CLI. diff --git a/content/rancher/v2.6/en/installation/install-rancher-on-k8s/_index.md b/content/rancher/v2.6/en/installation/install-rancher-on-k8s/_index.md index 11b5f6c2def..5d97210cc95 100644 --- a/content/rancher/v2.6/en/installation/install-rancher-on-k8s/_index.md +++ b/content/rancher/v2.6/en/installation/install-rancher-on-k8s/_index.md @@ -17,7 +17,7 @@ In this section, you'll learn how to deploy Rancher on a Kubernetes cluster usin ### Kubernetes Cluster -Set up the Rancher server's local Kubernetes cluster. +Set up the Rancher server's local Kubernetes cluster. Rancher can be installed on any Kubernetes cluster. This cluster can use upstream Kubernetes, or it can use one of Rancher's Kubernetes distributions, or it can be a managed Kubernetes cluster from a provider such as Amazon EKS. @@ -104,8 +104,12 @@ There are three recommended options for the source of the certificate used for T ### 4. Install cert-manager +**Note:** New in v2.6.4, cert-manager versions 1.6.2 and 1.7.1 are compatible. We recommend v1.7.x because v 1.6.x will reach end-of-life on March 30, 2022. + > You should skip this step if you are bringing your own certificate files (option `ingress.tls.source=secret`), or if you use [TLS termination on an external load balancer]({{}}/rancher/v2.6/en/installation/install-rancher-on-k8s/chart-options/#external-tls-termination). +> You should skip this step if you are bringing your own certificate files (option `ingress.tls.source=secret`), or if you use [TLS termination on an external load balancer]({{}}/rancher/v2.6/en/installation/install-rancher-on-k8s/chart-options/#external-tls-termination). + This step is only required to use certificates issued by Rancher's generated CA (`ingress.tls.source=rancher`) or to request Let's Encrypt issued certificates (`ingress.tls.source=letsEncrypt`). {{% accordion id="cert-manager" label="Click to Expand" %}} @@ -116,7 +120,7 @@ These instructions are adapted from the [official cert-manager documentation](ht ``` # If you have installed the CRDs manually instead of with the `--set installCRDs=true` option added to your Helm install command, you should upgrade your CRD resources before upgrading the Helm chart: -kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.5.1/cert-manager.crds.yaml +kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.7.1/cert-manager.crds.yaml # Add the Jetstack Helm repository helm repo add jetstack https://charts.jetstack.io @@ -128,7 +132,7 @@ helm repo update helm install cert-manager jetstack/cert-manager \ --namespace cert-manager \ --create-namespace \ - --version v1.5.1 + --version v1.7.1 ``` Once you’ve installed cert-manager, you can verify it is deployed correctly by checking the cert-manager namespace for running pods: @@ -148,6 +152,10 @@ cert-manager-webhook-787858fcdb-nlzsq 1/1 Running 0 2m The exact command to install Rancher differs depending on the certificate configuration. +However, irrespective of the certificate configuration, the name of the Rancher installation in the `cattle-system` namespace should always be `rancher`. + +> **Tip for testing and development:** This final command to install Rancher requires a domain name that forwards traffic to Rancher. If you are using the Helm CLI to set up a proof-of-concept, you can use a fake domain name when passing the `hostname` option. An example of a fake domain name would be `.sslip.io`, which would expose Rancher on an IP where it is running. Production installs would require a real domain name. + {{% tabs %}} {{% tab "Rancher-generated Certificates" %}} @@ -158,7 +166,7 @@ Because `rancher` is the default option for `ingress.tls.source`, we are not spe - Set the `hostname` to the DNS name you pointed at your load balancer. - Set the `bootstrapPassword` to something unique for the `admin` user. -- If you are installing an alpha version, Helm requires adding the `--devel` option to the command. +- If you are installing an alpha version, Helm requires adding the `--devel` option to the command. - To install a specific Rancher version, use the `--version` flag, example: `--version 2.3.6` ``` @@ -181,7 +189,7 @@ deployment "rancher" successfully rolled out This option uses `cert-manager` to automatically request and renew [Let's Encrypt](https://letsencrypt.org/) certificates. This is a free service that provides you with a valid certificate as Let's Encrypt is a trusted CA. ->**Note:**: You need to have port 80 open as the HTTP-01 challenge can only be done on port 80. +>**Note:** You need to have port 80 open as the HTTP-01 challenge can only be done on port 80. In the following command, @@ -189,7 +197,8 @@ In the following command, - Set the `bootstrapPassword` to something unique for the `admin` user. - `ingress.tls.source` is set to `letsEncrypt` - `letsEncrypt.email` is set to the email address used for communication about your certificate (for example, expiry notices) -- If you are installing an alpha version, Helm requires adding the `--devel` option to the command. +- Set `letsEncrypt.ingress.class` to whatever your ingress controller is, e.g., `traefik`, `nginx`, `haproxy`, etc. +- If you are installing an alpha version, Helm requires adding the `--devel` option to the command. ``` helm install rancher rancher-/rancher \ @@ -197,7 +206,8 @@ helm install rancher rancher-/rancher \ --set hostname=rancher.my.org \ --set bootstrapPassword=admin \ --set ingress.tls.source=letsEncrypt \ - --set letsEncrypt.email=me@example.org + --set letsEncrypt.email=me@example.org \ + --set letsEncrypt.ingress.class=nginx ``` Wait for Rancher to be rolled out: @@ -221,7 +231,7 @@ Although an entry in the `Subject Alternative Names` is technically required, ha - Set the `hostname`. - Set the `bootstrapPassword` to something unique for the `admin` user. - Set `ingress.tls.source` to `secret`. -- If you are installing an alpha version, Helm requires adding the `--devel` option to the command. +- If you are installing an alpha version, Helm requires adding the `--devel` option to the command. ``` helm install rancher rancher-/rancher \ @@ -234,7 +244,7 @@ helm install rancher rancher-/rancher \ If you are using a Private CA signed certificate , add `--set privateCA=true` to the command: ``` -helm install rancher rancher-latest/rancher \ +helm install rancher rancher-/rancher \ --namespace cattle-system \ --set hostname=rancher.my.org \ --set bootstrapPassword=admin \ diff --git a/content/rancher/v2.6/en/installation/install-rancher-on-k8s/chart-options/_index.md b/content/rancher/v2.6/en/installation/install-rancher-on-k8s/chart-options/_index.md index 9ce31bc66f9..4436fcfe473 100644 --- a/content/rancher/v2.6/en/installation/install-rancher-on-k8s/chart-options/_index.md +++ b/content/rancher/v2.6/en/installation/install-rancher-on-k8s/chart-options/_index.md @@ -137,7 +137,7 @@ To customize or use a different ingress with Rancher server you can set your own Example on setting a custom certificate issuer: ```plain ---set ingress.extraAnnotations.'certmanager\.k8s\.io/cluster-issuer'=ca-key-pair +--set ingress.extraAnnotations.'cert-manager\.io/cluster-issuer'=issuer-name ``` Example on setting a static proxy header with `ingress.configurationSnippet`. This value is parsed like a template so variables can be used. diff --git a/content/rancher/v2.6/en/installation/install-rancher-on-k8s/upgrades/_index.md b/content/rancher/v2.6/en/installation/install-rancher-on-k8s/upgrades/_index.md index 562a1d8223b..cfe481e2b94 100644 --- a/content/rancher/v2.6/en/installation/install-rancher-on-k8s/upgrades/_index.md +++ b/content/rancher/v2.6/en/installation/install-rancher-on-k8s/upgrades/_index.md @@ -148,7 +148,13 @@ helm upgrade rancher rancher-/rancher \ ### Option B: Reinstalling Rancher and cert-manager -If you are currently running the cert-manger whose version is older than v0.11, and want to upgrade both Rancher and cert-manager to a newer version, then you need to reinstall both Rancher and cert-manger due to the API change in cert-manger v0.11. +If you are currently running the cert-manager whose version is 1.5 or below, and want to upgrade both Rancher and cert-manager to a new version (1.6+ in the case of cert-manager), then you need to re-install both Rancher and cert-manager due to the API change in cert-manager 1.6. This will also be necessary if you are upgrading from a version of cert manager below 0.11 to a version of cert-manager above 0.11. + +>**Important:** +> +>- New in v2.6.4, cert-manager versions 1.6.2 and 1.7.1 are compatible. We recommend v1.7.x because v 1.6.x will reach end-of-life on March 30, 2022. +> +>- Note that if you are below version 1.5 and want to go to 1.7, you should first upgrade to 1.6 as an intermediate step. Follow the cert-manager docs [here](https://cert-manager.io/docs/installation/upgrading/upgrading-1.5-1.6/) to do a 1.5 to 1.6 upgrade, and [here](https://cert-manager.io/docs/installation/upgrading/upgrading-1.6-1.7/) to then do a 1.6 to 1.7 upgrade. For more details on upgrading cert-manager, refer to our [documentation]({{}}/rancher/v2.6/en/installation/resources/upgrading-cert-manager). 1. Uninstall Rancher diff --git a/content/rancher/v2.6/en/installation/other-installation-methods/air-gap/install-rancher/_index.md b/content/rancher/v2.6/en/installation/other-installation-methods/air-gap/install-rancher/_index.md index d2be9a4a1ef..a739fd07987 100644 --- a/content/rancher/v2.6/en/installation/other-installation-methods/air-gap/install-rancher/_index.md +++ b/content/rancher/v2.6/en/installation/other-installation-methods/air-gap/install-rancher/_index.md @@ -94,8 +94,10 @@ helm repo update Fetch the latest cert-manager chart available from the [Helm chart repository](https://hub.helm.sh/charts/jetstack/cert-manager). +**Note:** New in v2.6.4, cert-manager versions 1.6.2 and 1.7.1 are compatible. We recommend v1.7.x because v 1.6.x will reach end-of-life on March 30, 2022. + ```plain -helm fetch jetstack/cert-manager --version v1.5.1 +helm fetch jetstack/cert-manager --version v1.7.1 ``` ### 3. Render the cert-manager template @@ -103,18 +105,19 @@ helm fetch jetstack/cert-manager --version v1.5.1 Render the cert-manager template with the options you would like to use to install the chart. Remember to set the `image.repository` option to pull the image from your private registry. This will create a `cert-manager` directory with the Kubernetes manifest files. ```plain -helm template cert-manager ./cert-manager-v1.5.1.tgz --output-dir . \ +helm template cert-manager ./cert-manager-v1.7.1.tgz --output-dir . \ --namespace cert-manager \ --set image.repository=/quay.io/jetstack/cert-manager-controller \ --set webhook.image.repository=/quay.io/jetstack/cert-manager-webhook \ - --set cainjector.image.repository=/quay.io/jetstack/cert-manager-cainjector + --set cainjector.image.repository=/quay.io/jetstack/cert-manager-cainjector \ + --set startupapicheck.image.repository=/quay.io/jetstack/cert-manager-ctl ``` ### 4. Download the cert-manager CRD Download the required CRD file for cert-manager: ```plain - curl -L -o cert-manager/cert-manager-crd.yaml https://github.com/jetstack/cert-manager/releases/download/v1.5.1/cert-manager.crds.yaml + curl -L -o cert-manager/cert-manager-crd.yaml https://github.com/jetstack/cert-manager/releases/download/v1.7.1/cert-manager.crds.yaml ``` ### 5. Render the Rancher template diff --git a/content/rancher/v2.6/en/installation/other-installation-methods/air-gap/populate-private-registry/_index.md b/content/rancher/v2.6/en/installation/other-installation-methods/air-gap/populate-private-registry/_index.md index 2c27c28bb2c..18c8817f76f 100644 --- a/content/rancher/v2.6/en/installation/other-installation-methods/air-gap/populate-private-registry/_index.md +++ b/content/rancher/v2.6/en/installation/other-installation-methods/air-gap/populate-private-registry/_index.md @@ -51,6 +51,8 @@ If you will use ARM64 hosts, the registry must support manifests. As of April 20 In a Kubernetes Install, if you elect to use the Rancher default self-signed TLS certificates, you must add the [`cert-manager`](https://hub.helm.sh/charts/jetstack/cert-manager) image to `rancher-images.txt` as well. +**Note:** New in v2.6.4, cert-manager versions 1.6.2 and 1.7.1 are compatible. We recommend v1.7.x because v 1.6.x will reach end-of-life on March 30, 2022. + 1. Fetch the latest `cert-manager` Helm chart and parse the template for image details: > **Note:** Recent changes to cert-manager require an upgrade. If you are upgrading Rancher and using a version of cert-manager older than v0.12.0, please see our [upgrade documentation]({{}}/rancher/v2.6/en/installation/resources/upgrading-cert-manager/). @@ -58,8 +60,8 @@ In a Kubernetes Install, if you elect to use the Rancher default self-signed TLS ```plain helm repo add jetstack https://charts.jetstack.io helm repo update - helm fetch jetstack/cert-manager --version v1.5.1 - helm template ./cert-manager-.tgz | grep -oP '(?<=image: ").*(?=")' >> ./rancher-images.txt + helm fetch jetstack/cert-manager --version v1.7.1 + helm template ./cert-manager-.tgz | awk '$1 ~ /image:/ {print $2}' | sed s/\"//g >> ./rancher-images.txt ``` 2. Sort and unique the images list to remove any overlap between the sources: @@ -230,7 +232,7 @@ The workstation must have Docker 18.02+ in order to support manifests, which are helm repo add jetstack https://charts.jetstack.io helm repo update helm fetch jetstack/cert-manager --version v0.12.0 - helm template ./cert-manager-.tgz | grep -oP '(?<=image: ").*(?=")' >> ./rancher-images.txt + helm template ./cert-manager-.tgz | awk '$1 ~ /image:/ {print $2}' | sed s/\"//g >> ./rancher-images.txt ``` 2. Sort and unique the images list to remove any overlap between the sources: diff --git a/content/rancher/v2.6/en/installation/other-installation-methods/air-gap/prepare-nodes/_index.md b/content/rancher/v2.6/en/installation/other-installation-methods/air-gap/prepare-nodes/_index.md index e054670c0a0..19cc8159774 100644 --- a/content/rancher/v2.6/en/installation/other-installation-methods/air-gap/prepare-nodes/_index.md +++ b/content/rancher/v2.6/en/installation/other-installation-methods/air-gap/prepare-nodes/_index.md @@ -142,7 +142,7 @@ For a how-to guide for setting up a DNS record to route domain traffic to an Ama Rancher supports air gap installs using a secure Docker private registry. You must have your own private registry or other means of distributing Docker images to your machines. -In a later step, when you set up your K3s Kubernetes cluster, you will create a [private registries configuration file]({{}}/k3s/latest/en/installation/private-registry/) with details from this registry. +In a later step, when you set up your RKE Kubernetes cluster, you will create a [private registries configuration file]({{}}/rke/latest/en/config-options/private-registries/) with details from this registry. If you need help with creating a private registry, please refer to the [official Docker documentation.](https://docs.docker.com/registry/deploying/#run-an-externally-accessible-registry) @@ -164,8 +164,6 @@ For an example of one way to set up Linux nodes, refer to this [tutorial]({{}}/k3s/latest/en/installation/private-registry/) with details from this registry. - If you need help with creating a private registry, please refer to the [official Docker documentation.](https://docs.docker.com/registry/) {{% /tab %}} diff --git a/content/rancher/v2.6/en/installation/other-installation-methods/behind-proxy/install-rancher/_index.md b/content/rancher/v2.6/en/installation/other-installation-methods/behind-proxy/install-rancher/_index.md index ae9faf9304b..c147692a558 100644 --- a/content/rancher/v2.6/en/installation/other-installation-methods/behind-proxy/install-rancher/_index.md +++ b/content/rancher/v2.6/en/installation/other-installation-methods/behind-proxy/install-rancher/_index.md @@ -23,15 +23,17 @@ kubectl create namespace cert-manager Install the CustomResourceDefinitions of cert-manager: +**Note:** New in v2.6.4, cert-manager versions 1.6.2 and 1.7.1 are compatible. We recommend v1.7.x because v 1.6.x will reach end-of-life on March 30, 2022. + ``` -kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.5.1/cert-manager.crds.yaml +kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.7.1/cert-manager.crds.yaml ``` And install it with Helm. Note that cert-manager also needs your proxy configured in case it needs to communicate with Let's Encrypt or other external certificate issuers: ``` helm upgrade --install cert-manager jetstack/cert-manager \ - --namespace cert-manager --version v1.5.1 \ + --namespace cert-manager --version v1.7.1 \ --set http_proxy=http://${proxy_host} \ --set https_proxy=http://${proxy_host} \ --set no_proxy=127.0.0.0/8\\,10.0.0.0/8\\,cattle-system.svc\\,172.16.0.0/12\\,192.168.0.0/16\\,.svc\\,.cluster.local @@ -64,8 +66,8 @@ And install Rancher with Helm. Rancher also needs a proxy configuration so that helm upgrade --install rancher rancher-latest/rancher \ --namespace cattle-system \ --set hostname=rancher.example.com \ - --set proxy=http://${proxy_host} - --set no_proxy=127.0.0.0/8\\,10.0.0.0/8\\,cattle-system.svc\\,172.16.0.0/12\\,192.168.0.0/16\\,.svc\\,.cluster.local + --set proxy=http://${proxy_host} \ + --set noProxy=127.0.0.0/8\\,10.0.0.0/8\\,cattle-system.svc\\,172.16.0.0/12\\,192.168.0.0/16\\,.svc\\,.cluster.local ``` After waiting for the deployment to finish: diff --git a/content/rancher/v2.6/en/installation/other-installation-methods/behind-proxy/launch-kubernetes/_index.md b/content/rancher/v2.6/en/installation/other-installation-methods/behind-proxy/launch-kubernetes/_index.md index b47128310e9..41428448b27 100644 --- a/content/rancher/v2.6/en/installation/other-installation-methods/behind-proxy/launch-kubernetes/_index.md +++ b/content/rancher/v2.6/en/installation/other-installation-methods/behind-proxy/launch-kubernetes/_index.md @@ -9,7 +9,7 @@ Once the infrastructure is ready, you can continue with setting up an RKE cluste First, you have to install Docker and setup the HTTP proxy on all three Linux nodes. For this perform the following steps on all three nodes. -For convenience export the IP address and port of your proxy into an environment variable and set up the HTTP_PROXY variables for your current shell: +For convenience, export the IP address and port of your proxy into an environment variable and set up the HTTP_PROXY variables for your current shell: ``` export proxy_host="10.0.0.5:8888" @@ -58,6 +58,24 @@ sudo systemctl daemon-reload sudo systemctl restart docker ``` +#### Air-gapped proxy + +_New in v2.6.4_ + +You can now provision node driver clusters from an air-gapped cluster configured to use a proxy for outbound connections. + +In addition to setting the default rules for a proxy server, you will need to add additional rules, shown below, to provision node driver clusters from a proxied Rancher environment. + +You will configure your filepath according to your setup, e.g., `/etc/apt/apt.conf.d/proxy.conf`: + +``` +acl SSL_ports port 22 +acl SSL_ports port 2376 + +acl Safe_ports port 22 # ssh +acl Safe_ports port 2376 # docker port +``` + ### Creating the RKE Cluster You need several command line tools on the host where you have SSH access to the Linux nodes to create and interact with the cluster: diff --git a/content/rancher/v2.6/en/installation/other-installation-methods/single-node-docker/_index.md b/content/rancher/v2.6/en/installation/other-installation-methods/single-node-docker/_index.md index c047052f024..a52c192fe83 100644 --- a/content/rancher/v2.6/en/installation/other-installation-methods/single-node-docker/_index.md +++ b/content/rancher/v2.6/en/installation/other-installation-methods/single-node-docker/_index.md @@ -44,14 +44,13 @@ Choose from the following options: - [Option B: Bring Your Own Certificate, Self-signed](#option-b-bring-your-own-certificate-self-signed) - [Option C: Bring Your Own Certificate, Signed by a Recognized CA](#option-c-bring-your-own-certificate-signed-by-a-recognized-ca) - [Option D: Let's Encrypt Certificate](#option-d-let-s-encrypt-certificate) +- [Option E: Localhost tunneling, no Certificate](#option-e-localhost-tunneling-no-certificate) ### Option A: Default Rancher-generated Self-signed Certificate If you are installing Rancher in a development or testing environment where identity verification isn't a concern, install Rancher using the self-signed certificate that it generates. This installation option omits the hassle of generating a certificate yourself. -Log into your Linux host, and then run the minimum installation command below. - -Privileged access is [required.](#privileged-access-for-rancher) +Log into your host, and run the command below: ```bash docker run -d --restart=unless-stopped \ @@ -78,7 +77,7 @@ After creating your certificate, run the Docker command below to install Rancher | `` | The path to the private key for your certificate. | | `` | The path to the certificate authority's certificate. | -Privileged access is [required.](#privileged-access-for-rancher) +Log into your host, and run the command below: ```bash docker run -d --restart=unless-stopped \ @@ -92,7 +91,9 @@ docker run -d --restart=unless-stopped \ ### Option C: Bring Your Own Certificate, Signed by a Recognized CA -In production environments where you're exposing an app publicly, use a certificate signed by a recognized CA so that your user base doesn't encounter security warnings. +In production environments where you're exposing an app publicly, you would use a certificate signed by a recognized CA so that your user base doesn't encounter security warnings. + +The Docker install is not recommended for production. These instructions are provided for testing and development purposes only. > **Prerequisites:** > @@ -110,7 +111,7 @@ After obtaining your certificate, run the Docker command below. | `` | The path to your full certificate chain. | | `` | The path to the private key for your certificate. | -Privileged access is [required.](#privileged-access-for-rancher) +Log into your host, and run the command below: ```bash docker run -d --restart=unless-stopped \ @@ -128,6 +129,8 @@ docker run -d --restart=unless-stopped \ For production environments, you also have the option of using [Let's Encrypt](https://letsencrypt.org/) certificates. Let's Encrypt uses an http-01 challenge to verify that you have control over your domain. You can confirm that you control the domain by pointing the hostname that you want to use for Rancher access (for example, `rancher.mydomain.com`) to the IP of the machine it is running on. You can bind the hostname to the IP address by creating an A record in DNS. +The Docker install is not recommended for production. These instructions are provided for testing and development purposes only. + > **Prerequisites:** > > - Let's Encrypt is an Internet service. Therefore, this option cannot be used in an internal/air gapped network. @@ -140,7 +143,7 @@ After you fulfill the prerequisites, you can install Rancher using a Let's Encry | ----------------- | ------------------- | | `` | Your domain address | -Privileged access is [required.](#privileged-access-for-rancher) +Log into your host, and run the command below: ``` docker run -d --restart=unless-stopped \ @@ -150,6 +153,22 @@ docker run -d --restart=unless-stopped \ --acme-domain ``` +### Option E: Localhost tunneling, no Certificate + +If you are installing Rancher in a development or testing environment where you have a localhost tunneling solution running, such as [ngrok](https://ngrok.com/), avoid generating a certificate. This installation option doesn't require a certificate. + +- You will use `--no-cacerts` in the argument to disable the default CA certificate generated by Rancher. + +Log into your host, and run the command below: + +```bash +docker run -d --restart=unless-stopped \ + -p 80:80 -p 443:443 \ + --privileged \ + rancher/rancher:latest \ + --no-cacerts +``` + ## Advanced Options When installing Rancher on a single node with Docker, there are several advanced options that can be enabled: diff --git a/content/rancher/v2.6/en/installation/other-installation-methods/single-node-docker/proxy/_index.md b/content/rancher/v2.6/en/installation/other-installation-methods/single-node-docker/proxy/_index.md index 1ac4a66c017..37d3d2cfc00 100644 --- a/content/rancher/v2.6/en/installation/other-installation-methods/single-node-docker/proxy/_index.md +++ b/content/rancher/v2.6/en/installation/other-installation-methods/single-node-docker/proxy/_index.md @@ -40,3 +40,21 @@ docker run -d --restart=unless-stopped \ ``` Privileged access is [required.]({{}}/rancher/v2.6/en/installation/other-installation-methods/single-node-docker/#privileged-access-for-rancher) + +### Air-gapped proxy configuration + +_New in v2.6.4_ + +You can now provision node driver clusters from an air-gapped cluster configured to use a proxy for outbound connections. + +In addition to setting the default rules for a proxy server as shown above, you will need to add additional rules, shown below, to provision node driver clusters from a proxied Rancher environment. + +You will configure your filepath according to your setup, e.g., `/etc/apt/apt.conf.d/proxy.conf`: + +``` +acl SSL_ports port 22 +acl SSL_ports port 2376 + +acl Safe_ports port 22 # ssh +acl Safe_ports port 2376 # docker port +``` \ No newline at end of file diff --git a/content/rancher/v2.6/en/installation/other-installation-methods/single-node-docker/single-node-upgrades/_index.md b/content/rancher/v2.6/en/installation/other-installation-methods/single-node-docker/single-node-upgrades/_index.md index 7ee1d1f9328..3b03de02a22 100644 --- a/content/rancher/v2.6/en/installation/other-installation-methods/single-node-docker/single-node-upgrades/_index.md +++ b/content/rancher/v2.6/en/installation/other-installation-methods/single-node-docker/single-node-upgrades/_index.md @@ -5,9 +5,11 @@ weight: 1010 The following instructions will guide you through upgrading a Rancher server that was installed with Docker. +> **Docker installs are not supported in production environments.** These instructions are provided for testing and development purposes only. If you have already deployed a Docker install in production and need to upgrade to a new Rancher version, we recommend [migrating to the Helm chart install]({{}}/rancher/v2.6/en/backups/migrating-rancher/) before upgrading. + # Prerequisites -- **Review the [known upgrade issues]({{}}/rancher/v2.6/en/installation/install-rancher-on-k8s/upgrades/#known-upgrade-issues) in the Rancher documentation for the most noteworthy issues to consider when upgrading Rancher. A more complete list of known issues for each Rancher version can be found in the release notes on [GitHub](https://github.com/rancher/rancher/releases) and on the [Rancher forums.](https://forums.rancher.com/c/announcements/12) Note that upgrades to or from any chart in the [rancher-alpha repository]({{}}/rancher/v2.6/en/installation/install-rancher-on-k8s/chart-options/#helm-chart-repositories/) aren’t supported. +- **Review the [known upgrade issues]({{}}/rancher/v2.6/en/installation/install-rancher-on-k8s/upgrades/#known-upgrade-issues)** section in the Rancher documentation for the most noteworthy issues to consider when upgrading Rancher. A more complete list of known issues for each Rancher version can be found in the release notes on [GitHub](https://github.com/rancher/rancher/releases) and on the [Rancher forums](https://forums.rancher.com/c/announcements/12). Note that upgrades to or from any chart in the [rancher-alpha repository]({{}}/rancher/v2.6/en/installation/install-rancher-on-k8s/chart-options/#helm-chart-repositories/) aren’t supported. - **For [air gap installs only,]({{}}/rancher/v2.6/en/installation/other-installation-methods/air-gap) collect and populate images for the new Rancher server version**. Follow the guide to [populate your private registry]({{}}/rancher/v2.6/en/installation/other-installation-methods/air-gap/populate-private-registry/) with the images for the Rancher version that you want to upgrade to. # Placeholder Review @@ -80,7 +82,7 @@ During upgrade, you create a copy of the data from your current Rancher containe ``` - docker run --volumes-from rancher-data -v $PWD:/backup busybox tar zcvf /backup/rancher-data-backup--.tar.gz /var/lib/rancher + docker run --volumes-from rancher-data -v "$PWD:/backup" --rm busybox tar zcvf /backup/rancher-data-backup--.tar.gz /var/lib/rancher ``` **Step Result:** When you enter this command, a series of commands should run. diff --git a/content/rancher/v2.6/en/installation/requirements/_index.md b/content/rancher/v2.6/en/installation/requirements/_index.md index 9f1c959f73b..f515e688580 100644 --- a/content/rancher/v2.6/en/installation/requirements/_index.md +++ b/content/rancher/v2.6/en/installation/requirements/_index.md @@ -68,7 +68,7 @@ net.bridge.bridge-nf-call-iptables=1 For the container runtime, K3s should work with any modern version of Docker or containerd. -Rancher needs to be installed on a supported Kubernetes version. To find out which versions of Kubernetes are supported for your Rancher version, refer to the [support maintenance terms.](https://rancher.com/support-maintenance-terms/) To specify the K3s version, use the INSTALL_K3S_VERSION environment variable when running the K3s installation script. +Rancher needs to be installed on a supported Kubernetes version. To find out which versions of Kubernetes are supported for your Rancher version, refer to the [support maintenance terms.](https://rancher.com/support-maintenance-terms/) To specify the K3s version, use the INSTALL_K3S_VERSION environment variable when running the K3s installation script. If you are installing Rancher on a K3s cluster with **Raspbian Buster**, follow [these steps]({{}}/k3s/latest/en/advanced/#enabling-legacy-iptables-on-raspbian-buster) to switch to legacy iptables. @@ -82,8 +82,6 @@ For details on which OS versions were tested with RKE2, refer to the [support ma Docker is not required for RKE2 installs. -The Ingress should be deployed as DaemonSet to ensure your load balancer can successfully route traffic to all nodes. Currently, RKE2 deploys nginx-ingress as a deployment by default, so you will need to deploy it as a DaemonSet by following [these steps.]({{}}/rancher/v2.6/en/installation/resources/k8s-tutorials/ha-rke2/#5-configure-nginx-to-be-a-daemonset) - ### Installing Docker Docker is required for Helm chart installs, and it can be installed by following the steps in the official [Docker documentation.](https://docs.docker.com/) Rancher also provides [scripts]({{}}/rancher/v2.6/en/installation/requirements/installing-docker) to install Docker with one command. @@ -110,7 +108,7 @@ These requirements apply to RKE Kubernetes clusters, as well as to hosted Kubern | X-Large | Up to 1000 | Up to 10,000 | 16 | 64 GB | | XX-Large | Up to 2000 | Up to 20,000 | 32 | 128 GB | -[Contact Rancher](https://rancher.com/contact/) for more than 2000 clusters and/or 20,000 nodes. +Every use case and environment is different. Please [contact Rancher](https://rancher.com/contact/) to review yours. ### K3s Kubernetes @@ -124,7 +122,7 @@ These CPU and memory requirements apply to each host in a [K3s Kubernetes cluste | X-Large | Up to 1000 | Up to 10,000 | 16 | 64 GB | 2 cores, 4 GB + 1000 IOPS | | XX-Large | Up to 2000 | Up to 20,000 | 32 | 128 GB | 2 cores, 4 GB + 1000 IOPS | -[Contact Rancher](https://rancher.com/contact/) for more than 2000 clusters and/or 20,000 nodes. +Every use case and environment is different. Please [contact Rancher](https://rancher.com/contact/) to review yours. ### RKE2 Kubernetes @@ -151,7 +149,7 @@ Each node in the Kubernetes cluster that Rancher is installed on should run an I The Ingress should be deployed as DaemonSet to ensure your load balancer can successfully route traffic to all nodes. -For RKE and K3s installations, you don't have to install the Ingress manually because is is installed by default. +For RKE and K3s installations, you don't have to install the Ingress manually because it is installed by default. For hosted Kubernetes clusters (EKS, GKE, AKS) and RKE2 Kubernetes installations, you will need to set up the ingress. @@ -170,6 +168,8 @@ Rancher performance depends on etcd in the cluster performance. To ensure optima This section describes the networking requirements for the node(s) where the Rancher server is installed. +> If a server containing Rancher has the `X-Frame-Options=DENY` header, some pages in the new Rancher UI will not be able to render after upgrading from the legacy UI. This is because some legacy pages are embedded as iFrames in the new UI. + ### Node IP Addresses Each node used should have a static IP configured, regardless of whether you are installing Rancher on a single node or on an HA cluster. In case of DHCP, each node should have a DHCP reservation to make sure the node gets the same IP allocated. @@ -180,12 +180,4 @@ To operate properly, Rancher requires a number of ports to be open on Rancher no # Dockershim Support -In Kubernetes v1.20, the dockershim became deprecated, and Docker became deprecated as a container runtime for Kubernetes. Dockershim was built into Kubernetes as a type of adapter that allowed Kubernetes to manage Docker containers. It was necessary because the Docker Daemon was not compliant with the CRI (Container Runtime Interface) that was created for Kubernetes. The dockershim is still included in the kubelet in Kubernetes v1.20. - -Rancher plans to implement the [upstream open source community Dockershim announced by Mirantis and Docker](https://www.mirantis.com/blog/mirantis-to-take-over-support-of-kubernetes-dockershim-2/) to ensure RKE clusters can continue to leverage Docker as their container runtime. Users of RKE will be able to continue upgrading and building new RKE clusters leveraging Docker as the runtime and install method. - -For users looking to use another container runtime, Rancher has the edge-focused K3s and datacenter-focused RKE2 Kubernetes distributions that use containerd as the default runtime. Imported RKE2 and K3s Kubernetes clusters can then be upgraded and managed through Rancher going forward. - -For more information on the deprecation of Docker as a container runtime for Kubernetes, see the [official Kubernetes blog post](https://kubernetes.io/blog/2020/12/02/dont-panic-kubernetes-and-docker/) and the [official blog post from Mirantis.](https://www.mirantis.com/blog/mirantis-to-take-over-support-of-kubernetes-dockershim-2/) - -The dockershim deprecation schedule is tracked by the upstream Kubernetes community in [Kubernetes Enhancement Proposal (KEP) 1985.](https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/1985-remove-dockershim) \ No newline at end of file +For more information on Dockershim support, refer to [this page]({{}}/rancher/v2.6/en/installation/requirements/dockershim/). diff --git a/content/rancher/v2.6/en/installation/requirements/dockershim/_index.md b/content/rancher/v2.6/en/installation/requirements/dockershim/_index.md new file mode 100644 index 00000000000..e4c3490c4b2 --- /dev/null +++ b/content/rancher/v2.6/en/installation/requirements/dockershim/_index.md @@ -0,0 +1,44 @@ +--- +title: Dockershim +weight: 300 +--- + +The Dockershim is the CRI compliant layer between the Kubelet and the Docker daemon. As part of the Kubernetes 1.20 release, the [deprecation of the in-tree Dockershim was announced](https://kubernetes.io/blog/2020/12/02/dont-panic-kubernetes-and-docker/). For more information on the deprecation and its timelines, see the [Kubernetes Dockershim Deprecation FAQ](https://kubernetes.io/blog/2020/12/02/dockershim-faq/#when-will-dockershim-be-removed). + +RKE clusters now support the external Dockershim to continue leveraging Docker as the CRI runtime. We now implement the upstream open source community external Dockershim announced by [Mirantis and Docker](https://www.mirantis.com/blog/mirantis-to-take-over-support-of-kubernetes-dockershim-2/) to ensure RKE clusters can continue to leverage Docker. + +To enable the external Dockershim, configure the following option. + +``` +enable_cri_dockerd: true +``` + +For users looking to use another container runtime, Rancher has the edge-focused K3s and datacenter-focused RKE2 Kubernetes distributions that use containerd as the default runtime. Imported RKE2 and K3s Kubernetes clusters can then be upgraded and managed through Rancher going forward. + +### FAQ + +
+ +Q. Do I have to upgrade Rancher to get Rancher’s support of the upstream Dockershim? + +A The upstream support of Dockershim begins for RKE in Kubernetes 1.21. You will need to be on a version of Rancher that supports RKE 1.21. See our support matrix for details. + +
+ +Q. I am currently on RKE with Kubernetes 1.20. Do I need to upgrade to RKE with Kubernetes 1.21 sooner to avoid being out of support for Dockershim? + +A. The version of Dockershim in RKE with Kubernetes 1.20 will continue to work and it is not deprecated until a later release. For information on the timeline, see the [Kubernetes Dockershim Deprecation FAQ](https://kubernetes.io/blog/2020/12/02/dockershim-faq/#when-will-dockershim-be-removed). It will only emit a warning of its future deprecation, which Rancher has mitigated in RKE with Kubernetes 1.21. You can plan your upgrade to 1.21 as you would normally. + +
+ +Q: What are my other options if I don’t want to depend on the Dockershim? + +A: You can use a runtime like containerd with Kubernetes that does not require Dockershim support. RKE2 or K3s are two options for doing this. + +
+ +Q: If I am already using RKE1 and want to switch to RKE2, what are my migration options? + +A: Today, you can stand up a new cluster and migrate workloads to a new RKE2 cluster that uses containerd. Rancher is exploring the possibility of an in-place upgrade path. + +
diff --git a/content/rancher/v2.6/en/installation/requirements/installing-docker/_index.md b/content/rancher/v2.6/en/installation/requirements/installing-docker/_index.md index 02a005d245f..8b16d157505 100644 --- a/content/rancher/v2.6/en/installation/requirements/installing-docker/_index.md +++ b/content/rancher/v2.6/en/installation/requirements/installing-docker/_index.md @@ -3,7 +3,7 @@ title: Installing Docker weight: 1 --- -For Helm CLI installs, Docker is required to be installed on any node that runs the Rancher server. +Docker is required to be installed on nodes where the Rancher server will be installed with Helm or Docker. There are a couple of options for installing Docker. One option is to refer to the [official Docker documentation](https://docs.docker.com/install/) about how to install Docker on Linux. The steps will vary based on the Linux distribution. diff --git a/content/rancher/v2.6/en/installation/requirements/ports/_index.md b/content/rancher/v2.6/en/installation/requirements/ports/_index.md index 9e724310864..a9eaaf1bd81 100644 --- a/content/rancher/v2.6/en/installation/requirements/ports/_index.md +++ b/content/rancher/v2.6/en/installation/requirements/ports/_index.md @@ -88,6 +88,7 @@ The following tables break down the port requirements for traffic between the Ra | TCP | 2379 | etcd client requests | | TCP | 2380 | etcd peer communication | | TCP | 6443 | Kubernetes apiserver | +| TCP | 8443 | Nginx Ingress's Validating Webhook | | UDP | 8472 | Canal/Flannel VXLAN overlay networking | | TCP | 9099 | Canal/Flannel livenessProbe/readinessProbe | | TCP | 10250 | Metrics server communication with all nodes | diff --git a/content/rancher/v2.6/en/installation/resources/advanced/firewall/_index.md b/content/rancher/v2.6/en/installation/resources/advanced/firewall/_index.md index 291cee6d594..69c1afae91d 100644 --- a/content/rancher/v2.6/en/installation/resources/advanced/firewall/_index.md +++ b/content/rancher/v2.6/en/installation/resources/advanced/firewall/_index.md @@ -3,7 +3,7 @@ title: Opening Ports with firewalld weight: 1 --- -> We recommend disabling firewalld. For Kubernetes 1.19, firewalld must be turned off. +> We recommend disabling firewalld. For Kubernetes 1.19.x and higher, firewalld must be turned off. Some distributions of Linux [derived from RHEL,](https://en.wikipedia.org/wiki/Red_Hat_Enterprise_Linux#Rebuilds) including Oracle Linux, may have default firewall rules that block communication with Helm. diff --git a/content/rancher/v2.6/en/installation/resources/feature-flags/_index.md b/content/rancher/v2.6/en/installation/resources/feature-flags/_index.md index a3fac147dad..25baf0c06a2 100644 --- a/content/rancher/v2.6/en/installation/resources/feature-flags/_index.md +++ b/content/rancher/v2.6/en/installation/resources/feature-flags/_index.md @@ -26,12 +26,12 @@ For example, if you install Rancher, then set a feature flag to true with the Ra The following is a list of the feature flags available in Rancher: - `harvester`: This feature flag is available starting in v2.6.1. It is used to manage access to the Virtualization Management page where users can navigate directly to Harvester clusters and access the Harvester UI. For more information, see [this page]({{}}/rancher/v2.6/en/virtualization-admin/#feature-flag/). -- `rke2`: We have introduced the ability to provision RKE2 clusters as tech preview. By default, this feature flag is enabled, which allows users to attempt to provision these type of clusters. +- `rke2`: Used to enable the ability to provision RKE2 clusters. By default, this feature flag is enabled, which allows users to attempt to provision these type of clusters. - `fleet`: The previous `fleet` feature flag is now required to be enabled as the Fleet capabilities are leveraged within the new provisioning framework. If you had this feature flag disabled in earlier versions, upon upgrading to Rancher v2.6, the flag will automatically be enabled. See this [page]({{}}/rancher/v2.6/en/deploy-across-clusters/fleet) for more information. - `continuous-delivery`: In Rancher v2.5.x, Fleet came with a GitOps feature that could not be disabled separately from Fleet. In Rancher v2.6, the `continuous-delivery` feature flag was introduced to allow the GitOps feature of Fleet to be disabled. For more information, see [this page.](./continuous-delivery). - `legacy`: There are a set of features from previous versions that are slowly being phased out of Rancher for newer iterations of the feature. This is a mix of deprecated features as well as features that will eventually be moved to newer variations in Rancher. By default, this feature flag is disabled for new installations. If you are upgrading from a previous version, this feature flag would be enabled. - `token-hashing`: Used to enable new token-hashing feature. Once enabled, existing tokens will be hashed and all new tokens will be hashed automatically using the SHA256 algorithm. Once a token is hashed it cannot be undone. Once this feature flag is enabled, it cannot be disabled. See [hashing of tokens]({{}}/rancher/v2.6/en/api/api-tokens) for more information. -- `unsupported-storage-drivers`: This feature [allows unsupported storage drivers.]({{}}/rancher/v2.6/en/installation/resources/feature-flags/enable-not-default-storage-drivers). In other words, it enables types for storage providers and provisioners that are not enabled by default. +- `unsupported-storage-drivers`: This feature [allows unsupported storage drivers.]({{}}/rancher/v2.6/en/installation/resources/feature-flags/enable-not-default-storage-drivers). In other words, it enables types for storage providers and provisioners that are not enabled by default. - `istio-virtual-service-ui`: This feature enables a [UI to create, read, update, and delete Istio virtual services and destination rules,]({{}}/rancher/v2.6/en/installation/resources/feature-flags/istio-virtual-service-ui) which are traffic management features of Istio. - `multi-cluster-management`: Used for multi-cluster provisioning and management of Kubernetes clusters. This feature flag can only be set at install time and not changed afterwards. @@ -61,7 +61,7 @@ When you install Rancher, enable the feature you want with a feature flag. The c > **Note:** Values set from the Rancher API will override the value passed in through the command line. -When installing Rancher with a Helm chart, use the `--features` option. In the below example, two features are enabled by passing the feature flag names names in a comma separated list: +When installing Rancher with a Helm chart, use the `--set` option. In the below example, two features are enabled by passing the feature flag names in a comma separated list: ``` helm install rancher rancher-latest/rancher \ @@ -102,7 +102,7 @@ When installing Rancher with Docker, use the `--features` option. In the below e docker run -d -p 80:80 -p 443:443 \ --restart=unless-stopped \ rancher/rancher:rancher-latest \ - --features==true,=true + --features==true,=true ``` diff --git a/content/rancher/v2.6/en/installation/resources/installing-docker/_index.md b/content/rancher/v2.6/en/installation/resources/installing-docker/_index.md deleted file mode 100644 index 69be5eb6baa..00000000000 --- a/content/rancher/v2.6/en/installation/resources/installing-docker/_index.md +++ /dev/null @@ -1,18 +0,0 @@ ---- -title: Installing Docker -weight: 1 ---- - -Docker is required to be installed on nodes where the Rancher server will be installed with Helm or Docker. - -There are a couple of options for installing Docker. One option is to refer to the [official Docker documentation](https://docs.docker.com/install/) about how to install Docker on Linux. The steps will vary based on the Linux distribution. - -Another option is to use one of Rancher's Docker installation scripts, which are available for most recent versions of Docker. - -For example, this command could be used to install Docker 19.03 on Ubuntu: - -``` -curl https://releases.rancher.com/install-docker/19.03.sh | sh -``` - -Rancher has installation scripts for every version of upstream Docker that Kubernetes supports. To find out whether a script is available for installing a certain Docker version, refer to this [GitHub repository,](https://github.com/rancher/install-docker) which contains all of Rancher's Docker installation scripts. \ No newline at end of file diff --git a/content/rancher/v2.6/en/installation/resources/k8s-tutorials/ha-RKE/_index.md b/content/rancher/v2.6/en/installation/resources/k8s-tutorials/ha-RKE/_index.md index 65ab4409c3d..ce46ab810c0 100644 --- a/content/rancher/v2.6/en/installation/resources/k8s-tutorials/ha-RKE/_index.md +++ b/content/rancher/v2.6/en/installation/resources/k8s-tutorials/ha-RKE/_index.md @@ -80,6 +80,8 @@ ingress: > Please see the [RKE Documentation]({{}}/rke/latest/en/config-options/) for the full list of options and capabilities. > > For tuning your etcd cluster for larger Rancher installations, see the [etcd settings guide]({{}}/rancher/v2.6/en/installation/resources/advanced/etcd/). +> +> For more information regarding Dockershim support, refer to [this page]({{}}/rancher/v2.6/en/installation/requirements/dockershim/) ### 2. Run RKE diff --git a/content/rancher/v2.6/en/installation/resources/k8s-tutorials/ha-rke2/_index.md b/content/rancher/v2.6/en/installation/resources/k8s-tutorials/ha-rke2/_index.md index 6c72adfaf09..f69c494b175 100644 --- a/content/rancher/v2.6/en/installation/resources/k8s-tutorials/ha-rke2/_index.md +++ b/content/rancher/v2.6/en/installation/resources/k8s-tutorials/ha-rke2/_index.md @@ -5,7 +5,7 @@ weight: 2 --- _Tested on v2.5.6_ -This section describes how to install a Kubernetes cluster according to the [best practices for the Rancher server environment.]({{}}/rancher/v2.x/en/overview/architecture-recommendations/#environment-for-kubernetes-installations) +This section describes how to install a Kubernetes cluster according to the [best practices for the Rancher server environment.]({{}}/rancher/v2.6/en/overview/architecture-recommendations/#environment-for-kubernetes-installations) # Prerequisites @@ -89,7 +89,7 @@ To use this `kubeconfig` file, 1. Install [kubectl,](https://kubernetes.io/docs/tasks/tools/install-kubectl/#install-kubectl) a Kubernetes command-line tool. 2. Copy the file at `/etc/rancher/rke2/rke2.yaml` and save it to the directory `~/.kube/config` on your local machine. -3. In the kubeconfig file, the `server` directive is defined as localhost. Configure the server as the DNS of your load balancer, referring to port 6443. (The Kubernetes API server will be reached at port 6443, while the Rancher server will be reached at ports 80 and 443.) Here is an example `rke2.yaml`: +3. In the kubeconfig file, the `server` directive is defined as localhost. Configure the server as the DNS of your control-plane load balancer, on port 6443. (The RKE2 Kubernetes API Server uses port 6443, while the Rancher server will be served via the NGINX Ingress on ports 80 and 443.) Here is an example `rke2.yaml`: ```yml apiVersion: v1 @@ -128,55 +128,40 @@ Now that you have set up the `kubeconfig` file, you can use `kubectl` to access Check that all the required pods and containers are healthy are ready to continue: ``` - /var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml get pods -A -NAMESPACE NAME READY STATUS RESTARTS AGE -kube-system etcd-ip-172-31-18-145 1/1 Running 0 4m37s -kube-system etcd-ip-172-31-25-73 1/1 Running 0 20m -kube-system etcd-ip-172-31-31-210 1/1 Running 0 9m12s -kube-system helm-install-rke2-canal-th9k9 0/1 Completed 0 21m -kube-system helm-install-rke2-coredns-6njr6 0/1 Completed 0 21m -kube-system helm-install-rke2-ingress-nginx-vztsd 0/1 Completed 0 21m -kube-system helm-install-rke2-kube-proxy-6std5 0/1 Completed 0 21m -kube-system helm-install-rke2-metrics-server-9sl7m 0/1 Completed 0 21m -kube-system kube-apiserver-ip-172-31-18-145 1/1 Running 0 4m22s -kube-system kube-apiserver-ip-172-31-25-73 1/1 Running 0 20m -kube-system kube-apiserver-ip-172-31-31-210 1/1 Running 0 9m8s -kube-system kube-controller-manager-ip-172-31-18-145 1/1 Running 0 4m8s -kube-system kube-controller-manager-ip-172-31-25-73 1/1 Running 0 21m -kube-system kube-controller-manager-ip-172-31-31-210 1/1 Running 0 8m55s -kube-system kube-proxy-57twm 1/1 Running 0 10m -kube-system kube-proxy-f7pc6 1/1 Running 0 5m24s -kube-system kube-proxy-rj4t5 1/1 Running 0 21m -kube-system kube-scheduler-ip-172-31-18-145 1/1 Running 0 4m15s -kube-system kube-scheduler-ip-172-31-25-73 1/1 Running 0 21m -kube-system kube-scheduler-ip-172-31-31-210 1/1 Running 0 8m48s -kube-system rke2-canal-4x972 2/2 Running 0 10m -kube-system rke2-canal-flh8m 2/2 Running 0 5m24s -kube-system rke2-canal-zfhkr 2/2 Running 0 21m -kube-system rke2-coredns-rke2-coredns-6cd96645d6-cmstq 1/1 Running 0 21m -kube-system rke2-ingress-nginx-controller-54946dd48f-6mp76 1/1 Running 0 20m -kube-system rke2-ingress-nginx-default-backend-5795954f8-p92xx 1/1 Running 0 20m -kube-system rke2-metrics-server-5f9b5757dc-k5sgh 1/1 Running 0 20m +/var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml get pods -A +NAMESPACE NAME READY STATUS RESTARTS AGE +kube-system cloud-controller-manager-rke2-server-1 1/1 Running 0 2m28s +kube-system cloud-controller-manager-rke2-server-2 1/1 Running 0 61s +kube-system cloud-controller-manager-rke2-server-3 1/1 Running 0 49s +kube-system etcd-rke2-server-1 1/1 Running 0 2m13s +kube-system etcd-rke2-server-2 1/1 Running 0 87s +kube-system etcd-rke2-server-3 1/1 Running 0 56s +kube-system helm-install-rke2-canal-hs6sx 0/1 Completed 0 2m17s +kube-system helm-install-rke2-coredns-xmzm8 0/1 Completed 0 2m17s +kube-system helm-install-rke2-ingress-nginx-flwnl 0/1 Completed 0 2m17s +kube-system helm-install-rke2-metrics-server-7sggn 0/1 Completed 0 2m17s +kube-system kube-apiserver-rke2-server-1 1/1 Running 0 116s +kube-system kube-apiserver-rke2-server-2 1/1 Running 0 66s +kube-system kube-apiserver-rke2-server-3 1/1 Running 0 48s +kube-system kube-controller-manager-rke2-server-1 1/1 Running 0 2m30s +kube-system kube-controller-manager-rke2-server-2 1/1 Running 0 57s +kube-system kube-controller-manager-rke2-server-3 1/1 Running 0 42s +kube-system kube-proxy-rke2-server-1 1/1 Running 0 2m25s +kube-system kube-proxy-rke2-server-2 1/1 Running 0 59s +kube-system kube-proxy-rke2-server-3 1/1 Running 0 85s +kube-system kube-scheduler-rke2-server-1 1/1 Running 0 2m30s +kube-system kube-scheduler-rke2-server-2 1/1 Running 0 57s +kube-system kube-scheduler-rke2-server-3 1/1 Running 0 42s +kube-system rke2-canal-b9lvm 2/2 Running 0 91s +kube-system rke2-canal-khwp2 2/2 Running 0 2m5s +kube-system rke2-canal-swfmq 2/2 Running 0 105s +kube-system rke2-coredns-rke2-coredns-547d5499cb-6tvwb 1/1 Running 0 92s +kube-system rke2-coredns-rke2-coredns-547d5499cb-rdttj 1/1 Running 0 2m8s +kube-system rke2-coredns-rke2-coredns-autoscaler-65c9bb465d-85sq5 1/1 Running 0 2m8s +kube-system rke2-ingress-nginx-controller-69qxc 1/1 Running 0 52s +kube-system rke2-ingress-nginx-controller-7hprp 1/1 Running 0 52s +kube-system rke2-ingress-nginx-controller-x658h 1/1 Running 0 52s +kube-system rke2-metrics-server-6564db4569-vdfkn 1/1 Running 0 66s ``` **Result:** You have confirmed that you can access the cluster with `kubectl` and the RKE2 cluster is running successfully. Now the Rancher management server can be installed on the cluster. - -### 5. Configure nginx to be a daemonset - -Currently, RKE2 deploys nginx-ingress as a deployment, and that can impact the Rancher deployment so that you cannot use all servers to proxy requests to the Rancher pods. - -To rectify that, place the following file in /var/lib/rancher/rke2/server/manifests on any of the server nodes: - -```yaml -apiVersion: helm.cattle.io/v1 -kind: HelmChartConfig -metadata: - name: rke2-ingress-nginx - namespace: kube-system -spec: - valuesContent: |- - controller: - kind: DaemonSet - daemonset: - useHostPort: true -``` diff --git a/content/rancher/v2.6/en/installation/resources/k8s-tutorials/infrastructure-tutorials/infra-for-rke2-ha/_index.md b/content/rancher/v2.6/en/installation/resources/k8s-tutorials/infrastructure-tutorials/infra-for-rke2-ha/_index.md index 82eb68522f7..e03f8ccc868 100644 --- a/content/rancher/v2.6/en/installation/resources/k8s-tutorials/infrastructure-tutorials/infra-for-rke2-ha/_index.md +++ b/content/rancher/v2.6/en/installation/resources/k8s-tutorials/infrastructure-tutorials/infra-for-rke2-ha/_index.md @@ -17,9 +17,9 @@ To install the Rancher management server on a high-availability RKE2 cluster, we ### 1. Set up Linux Nodes -Make sure that your nodes fulfill the general installation requirements for [OS, container runtime, hardware, and networking.]({{}}/rancher/v2.x/en/installation/requirements/) +Make sure that your nodes fulfill the general installation requirements for [OS, container runtime, hardware, and networking.]({{}}/rancher/v2.6/en/installation/requirements/) -For an example of one way to set up Linux nodes, refer to this [tutorial]({{}}/rancher/v2.x/en/installation/resources/k8s-tutorials/infrastructure-tutorials/ec2-node) for setting up nodes as instances in Amazon EC2. +For an example of one way to set up Linux nodes, refer to this [tutorial]({{}}/rancher/v2.6/en/installation/resources/k8s-tutorials/infrastructure-tutorials/ec2-node) for setting up nodes as instances in Amazon EC2. ### 2. Set up the Load Balancer @@ -32,11 +32,11 @@ When Rancher is installed (also in a later step), the Rancher system creates an For your implementation, consider if you want or need to use a Layer-4 or Layer-7 load balancer: - **A layer-4 load balancer** is the simpler of the two choices, in which you are forwarding TCP traffic to your nodes. We recommend configuring your load balancer as a Layer 4 balancer, forwarding traffic to ports TCP/80 and TCP/443 to the Rancher management cluster nodes. The Ingress controller on the cluster will redirect HTTP traffic to HTTPS and terminate SSL/TLS on port TCP/443. The Ingress controller will forward traffic to port TCP/80 to the Ingress pod in the Rancher deployment. -- **A layer-7 load balancer** is a bit more complicated but can offer features that you may want. For instance, a layer-7 load balancer is capable of handling TLS termination at the load balancer, as opposed to Rancher doing TLS termination itself. This can be beneficial if you want to centralize your TLS termination in your infrastructure. Layer-7 load balancing also offers the capability for your load balancer to make decisions based on HTTP attributes such as cookies, etc. that a layer-4 load balancer is not able to concern itself with. If you decide to terminate the SSL/TLS traffic on a layer-7 load balancer, you will need to use the `--set tls=external` option when installing Rancher in a later step. For more information, refer to the [Rancher Helm chart options.]({{}}/rancher/v2.x/en/installation/resources/chart-options/#external-tls-termination) +- **A layer-7 load balancer** is a bit more complicated but can offer features that you may want. For instance, a layer-7 load balancer is capable of handling TLS termination at the load balancer, as opposed to Rancher doing TLS termination itself. This can be beneficial if you want to centralize your TLS termination in your infrastructure. Layer-7 load balancing also offers the capability for your load balancer to make decisions based on HTTP attributes such as cookies, etc. that a layer-4 load balancer is not able to concern itself with. If you decide to terminate the SSL/TLS traffic on a layer-7 load balancer, you will need to use the `--set tls=external` option when installing Rancher in a later step. For more information, refer to the [Rancher Helm chart options.]({{}}/rancher/v2.6/en/installation/resources/chart-options/#external-tls-termination) For an example showing how to set up an NGINX load balancer, refer to [this page.]({{}}/rancher/v2.6/en/installation/resources/k8s-tutorials/infrastructure-tutorials/nginx/) -For a how-to guide for setting up an Amazon ELB Network Load Balancer, refer to [this page.]({{}}/rancher/v2.x/en/installation/resources/k8s-tutorials/infrastructure-tutorials/nlb/) +For a how-to guide for setting up an Amazon ELB Network Load Balancer, refer to [this page.]({{}}/rancher/v2.6/en/installation/resources/k8s-tutorials/infrastructure-tutorials/nlb/) > **Important:** > Do not use this load balancer (i.e, the `local` cluster Ingress) to load balance applications other than Rancher following installation. Sharing this Ingress with other applications may result in websocket errors to Rancher following Ingress configuration reloads for other apps. We recommend dedicating the `local` cluster to Rancher and no other applications. diff --git a/content/rancher/v2.6/en/installation/resources/tls-secrets/_index.md b/content/rancher/v2.6/en/installation/resources/tls-secrets/_index.md index e977c983669..cd11f1e9684 100644 --- a/content/rancher/v2.6/en/installation/resources/tls-secrets/_index.md +++ b/content/rancher/v2.6/en/installation/resources/tls-secrets/_index.md @@ -35,4 +35,4 @@ kubectl -n cattle-system create secret generic tls-ca \ # Updating a Private CA Certificate -Follow the steps on [this page]({{}}/rancher/v2.x/en/installation/resources/update-ca-cert) to update the SSL certificate of the ingress in a Rancher [high availability Kubernetes installation]({{}}/rancher/v2.6/en/installation/install-rancher-on-k8s/) or to switch from the default self-signed certificate to a custom certificate. \ No newline at end of file +Follow the steps on [this page]({{}}/rancher/v2.6/en/installation/resources/update-ca-cert) to update the SSL certificate of the ingress in a Rancher [high availability Kubernetes installation]({{}}/rancher/v2.6/en/installation/install-rancher-on-k8s/) or to switch from the default self-signed certificate to a custom certificate. \ No newline at end of file diff --git a/content/rancher/v2.6/en/installation/resources/update-ca-cert/_index.md b/content/rancher/v2.6/en/installation/resources/update-ca-cert/_index.md index 5a7a477fa82..057ca0f8c77 100644 --- a/content/rancher/v2.6/en/installation/resources/update-ca-cert/_index.md +++ b/content/rancher/v2.6/en/installation/resources/update-ca-cert/_index.md @@ -11,6 +11,7 @@ A summary of the steps is as follows: 2. Create or update the `tls-ca` Kubernetes secret resource with the root CA certificate (only required when using a private CA). 3. Update the Rancher installation using the Helm CLI. 4. Reconfigure the Rancher agents to trust the new CA certificate. +5. Select Force Update of Fleet clusters to connect fleet-agent to Rancher. The details of these instructions are below. @@ -132,7 +133,7 @@ Using a Kubeconfig for each downstream cluster update the environment variable f ``` $ kubectl edit -n cattle-system ds/cattle-node-agent -$ kubectl edit -n cattle-system deployment/cluster-agent +$ kubectl edit -n cattle-system deployment/cattle-cluster-agent ``` ### Method 3: Recreate Rancher agents @@ -143,3 +144,11 @@ First, generate the agent definitions as described here: https://gist.github.com Then, connect to a controlplane node of the downstream cluster via SSH, create a Kubeconfig and apply the definitions: https://gist.github.com/superseb/b14ed3b5535f621ad3d2aa6a4cd6443b + +# 5. Select Force Update of Fleet clusters to connect fleet-agent to Rancher + +Select 'Force Update' for the clusters within the [Continuous Delivery]({{}}/rancher/v2.6/en/deploy-across-clusters/fleet/#accessing-fleet-in-the-rancher-ui) view of the Rancher UI to allow the fleet-agent in downstream clusters to successfully connect to Rancher. + +### Why is this step required? + +Fleet agents in Rancher managed clusters store kubeconfig that is used to connect to the Rancher proxied kube-api in the fleet-agent secret of the fleet-system namespace. The kubeconfig contains a certificate-authority-data block containing the Rancher CA. When changing the Rancher CA, this block needs to be updated for a successful connection of the fleet-agent to Rancher. diff --git a/content/rancher/v2.6/en/installation/resources/upgrading-cert-manager/_index.md b/content/rancher/v2.6/en/installation/resources/upgrading-cert-manager/_index.md index da62f1e9deb..f3b21f8aed0 100644 --- a/content/rancher/v2.6/en/installation/resources/upgrading-cert-manager/_index.md +++ b/content/rancher/v2.6/en/installation/resources/upgrading-cert-manager/_index.md @@ -15,7 +15,7 @@ To address these changes, this guide will do two things: 1. Explain the cert-manager API changes and link to cert-manager's official documentation for migrating your data > **Important:** -> If you are currently running the cert-manger whose version is older than v0.11, and want to upgrade both Rancher and cert-manager to a newer version, you need to reinstall both of them: +> If you are currently running the cert-manager whose version is 1.5 or below, and want to upgrade both Rancher and cert-manager to a new version (1.6+ in the case of cert-manager), then you need to re-install both Rancher and cert-manager due to the API change in cert-manager 1.6. This will also be necessary if you are upgrading from a version of cert manager below 0.11 to a version of cert-manager above 0.11. Follow the steps below: > 1. Take a one-time snapshot of your Kubernetes cluster running Rancher server > 2. Uninstall Rancher, cert-manager, and the CustomResourceDefinition for cert-manager @@ -220,6 +220,13 @@ cert-manager-webhook-787858fcdb-nlzsq 1/1 Running 0 2m ## Cert-Manager API change and data migration +--- +_New in v2.6.4_ + +Rancher now supports cert-manager versions 1.6.2 and 1.7.1. We recommend v1.7.x because v 1.6.x will reach end-of-life on March 30, 2022. To read more, see the [cert-manager docs]({{}}/rancher/v2.6/en/installation/install-rancher-on-k8s/#4-install-cert-manager). For instructions on upgrading cert-manager from version 1.5 to 1.6, see the upstream cert-manager documentation [here](https://cert-manager.io/docs/installation/upgrading/upgrading-1.5-1.6/). For instructions on upgrading cert-manager from version 1.6 to 1.7, see the upstream cert-manager documentation [here](https://cert-manager.io/docs/installation/upgrading/upgrading-1.6-1.7/). + +--- + Cert-manager has deprecated the use of the `certificate.spec.acme.solvers` field and will drop support for it completely in an upcoming release. Per the cert-manager documentation, a new format for configuring ACME certificate resources was introduced in v0.8. Specifically, the challenge solver configuration field was moved. Both the old format and new are supported as of v0.9, but support for the old format will be dropped in an upcoming release of cert-manager. The cert-manager documentation strongly recommends that after upgrading you update your ACME Issuer and Certificate resources to the new format. diff --git a/content/rancher/v2.6/en/k8s-in-rancher/load-balancers-and-ingress/ingress/_index.md b/content/rancher/v2.6/en/k8s-in-rancher/load-balancers-and-ingress/ingress/_index.md index 3ca7dc768d4..bb69c198262 100644 --- a/content/rancher/v2.6/en/k8s-in-rancher/load-balancers-and-ingress/ingress/_index.md +++ b/content/rancher/v2.6/en/k8s-in-rancher/load-balancers-and-ingress/ingress/_index.md @@ -14,6 +14,7 @@ Ingresses can be added for workloads to provide load balancing, SSL termination 1. Enter a **Name** for the ingress. 1. Create ingress forwarding **Rules**. For help configuring the rules, refer to [this section.](#ingress-rule-configuration) If any of your ingress rules handle requests for encrypted ports, add a certificate to encrypt/decrypt communications. 1. **Optional:** click **Add Rule** to create additional ingress rules. For example, after you create ingress rules to direct requests for your hostname, you'll likely want to create a default backend to handle 404s. +1. Click **Create** at the bottom right. **Result:** Your ingress is added to the project. The ingress begins enforcing your ingress rules. diff --git a/content/rancher/v2.6/en/k8s-in-rancher/registries/_index.md b/content/rancher/v2.6/en/k8s-in-rancher/registries/_index.md index c5494e3d61e..0b075b620d0 100644 --- a/content/rancher/v2.6/en/k8s-in-rancher/registries/_index.md +++ b/content/rancher/v2.6/en/k8s-in-rancher/registries/_index.md @@ -14,13 +14,13 @@ Deployments use the Kubernetes registry secret to authenticate with a private Do Currently, deployments pull the private registry credentials automatically only if the workload is created in the Rancher UI and not when it is created via kubectl. -# Creating a Registry +# Creating a Registry in Namespaces >**Prerequisites:** You must have a [private registry](https://docs.docker.com/registry/deploying/) available to use. 1. In the upper left corner, click **☰ > Cluster Management**. 1. Go to the cluster where you want to add a registry and click **Explore**. -1. In the left navigation lick **More Resources > Core > Secrets**. +1. In the left navigation, click either **Storage > Secrets** or **More Resources > Core > Secrets**. 1. Click **Create**. 1. Click **Registry**. 1. Enter a **Name** for the registry. @@ -28,17 +28,46 @@ Currently, deployments pull the private registry credentials automatically only >**Note:** Kubernetes classifies secrets, certificates, and registries all as [secrets](https://kubernetes.io/docs/concepts/configuration/secret/), and no two secrets in a project or namespace can have duplicate names. Therefore, to prevent conflicts, your registry must have a unique name among all secrets within your workspace. 1. Select a namespace for the registry. - 1. Select the website that hosts your private registry. Then enter credentials that authenticate with the registry. For example, if you use DockerHub, provide your DockerHub username and password. - 1. Click **Save**. **Result:** - Your secret is added to the namespace you chose. -- You can view the secret in the Rancher UI from the **Resources > Registries** view. +- You can view the secret in the Rancher UI by clicking either **Storage > Secrets** or **More Resources > Core > Secrets**. - Any workload that you create in the Rancher UI will have the credentials to access the registry if the workload is within the registry's scope. +# Creating a Registry in Projects + +>**Prerequisites:** You must have a [private registry](https://docs.docker.com/registry/deploying/) available to use. + +Before v2.6, secrets were required to be in a project scope. Projects are no longer required, and you may use the namespace scope instead. As a result, the Rancher UI was updated to reflect this new functionality. However, you may still create a project-scoped registry if desired. Use the following steps to do so: + +1. In the upper left corner, click **☰ > Global Settings** in the dropdown. +1. Click **Feature Flags**. +1. Go to the `legacy` feature flag and click **Activate**. +1. In the upper left corner, click **☰ > Cluster Management**. +1. Go to the cluster where you want to add a registry and click **Explore**. +1. In the left navigation, click either **Storage > Secrets** or **More Resources > Core > Secrets**. +1. Click **Create**. +1. Click **Registry**. +1. In the top navigation bar, filter to see only one project. +1. Enter a **Name** for the registry. + + >**Note:** Kubernetes classifies secrets, certificates, and registries all as [secrets](https://kubernetes.io/docs/concepts/configuration/secret/), and no two secrets in a project or namespace can have duplicate names. Therefore, to prevent conflicts, your registry must have a unique name among all secrets within your workspace. + +1. Select a namespace for the registry. +1. Select the website that hosts your private registry. Then enter credentials that authenticate with the registry. For example, if you use DockerHub, provide your DockerHub username and password. +1. Click **Save**. + +**Result:** + +- Your secret is added to the individual project you chose. +- You can view the secret in the Rancher UI by clicking either **Storage > Secrets** or **More Resources > Core > Secrets**. +- Any workload that you create in the Rancher UI will have the credentials to access the registry if the workload is within the registry's scope. + +>**Note:** Project-scoped registries on the local cluster are only visible when a single project is selected. + # Using a Private Registry You can deploy a workload with an image from a private registry through the Rancher UI, or with `kubectl`. diff --git a/content/rancher/v2.6/en/k8s-in-rancher/secrets/_index.md b/content/rancher/v2.6/en/k8s-in-rancher/secrets/_index.md index 0663d4f1bc3..9c7f74220ea 100644 --- a/content/rancher/v2.6/en/k8s-in-rancher/secrets/_index.md +++ b/content/rancher/v2.6/en/k8s-in-rancher/secrets/_index.md @@ -11,17 +11,17 @@ When configuring a workload, you'll be able to choose which secrets to include. Mounted secrets will be updated automatically unless they are mounted as subpath volumes. For details on how updated secrets are propagated, refer to the [Kubernetes documentation.](https://kubernetes.io/docs/concepts/configuration/secret/#mounted-secrets-are-updated-automatically) -# Creating Secrets +# Creating Secrets in Namespaces 1. In the upper left corner, click **☰ > Cluster Management**. 1. Go to the cluster where you want to add a secret and click **Explore**. -1. Click **More Resources > Core > Secrets**. +1. To navigate to secrets, you may click either **Storage > Secrets** or **More Resources > Core > Secrets**. 1. Click **Create**. 1. Select the type of secret you want to create. 1. Select a **Namespace** for the secret. 1. Enter a **Name** for the secret. - >**Note:** Kubernetes classifies secrets, certificates, and registries all as [secrets](https://kubernetes.io/docs/concepts/configuration/secret/), and no two secrets in a project or namespace can have duplicate names. Therefore, to prevent conflicts, your secret must have a unique name among all secrets within your workspace. + >**Note:** Kubernetes classifies secrets, certificates, and registries all as [secrets](https://kubernetes.io/docs/concepts/configuration/secret/), and no two secrets in a namespace can have duplicate names. Therefore, to prevent conflicts, your secret must have a unique name among all secrets within your workspace. 1. From **Data**, click **Add** to add a key-value pair. Add as many values as you need. @@ -31,12 +31,31 @@ Mounted secrets will be updated automatically unless they are mounted as subpath 1. Click **Save**. -**Result:** Your secret is added to the project or namespace, depending on the scope you chose. You can view the secret in the Rancher UI from the **Resources > Secrets** view. +**Result:** Your secret is added to the namespace you chose. You can view the secret in the Rancher UI by clicking either **Storage > Secrets** or **More Resources > Core > Secrets**. Mounted secrets will be updated automatically unless they are mounted as subpath volumes. For details on how updated secrets are propagated, refer to the [Kubernetes documentation.](https://kubernetes.io/docs/concepts/configuration/secret/#mounted-secrets-are-updated-automatically) + +# Creating Secrets in Projects + +Before v2.6, secrets were required to be in a project scope. Projects are no longer required, and you may use the namespace scope instead. As a result, the Rancher UI was updated to reflect this new functionality. However, you may still create project-scoped secrets if desired. Note that you have to first enable the `legacy` feature flag and look at a single project to do so. Use the following steps to set up your project-level secret: + +1. In the upper left corner, click **☰ > Global Settings** in the dropdown. +1. Click **Feature Flags**. +1. Go to the `legacy` feature flag and click **Activate**. +1. In the upper left corner, click **☰ > Cluster Management** in the dropdown. +1. Go to the cluster that you created and click **Explore.** +1. Click **Legacy > Projects**. +1. In the top navigation bar, filter to see only one project. +1. In the left navigation bar, click **Secrets**. +1. Click **Add Secret**. + +**Result:** Your secret is added to the individual project you chose. You can view the secret in the Rancher UI by clicking either **Storage > Secrets** or **More Resources > Core > Secrets**. + +>**Note:** Project-scoped secrets on the local cluster are only visible when a single project is selected. + # What's Next? -Now that you have a secret added to the project or namespace, you can add it to a workload that you deploy. +Now that you have a secret added to a namespace, you can add it to a workload that you deploy. For more information on adding secret to a workload, see [Deploying Workloads]({{}}/rancher/v2.6/en/k8s-in-rancher/workloads/deploy-workloads/). diff --git a/content/rancher/v2.6/en/k8s-in-rancher/workloads/upgrade-workloads/_index.md b/content/rancher/v2.6/en/k8s-in-rancher/workloads/upgrade-workloads/_index.md index be8c09f7792..f6804adb2ea 100644 --- a/content/rancher/v2.6/en/k8s-in-rancher/workloads/upgrade-workloads/_index.md +++ b/content/rancher/v2.6/en/k8s-in-rancher/workloads/upgrade-workloads/_index.md @@ -16,6 +16,6 @@ When a new version of an application image is released on Docker Hub, you can up These options control how the upgrade rolls out to containers that are currently running. For example, for scalable deployments, you can choose whether you want to stop old pods before deploying new ones, or vice versa, as well as the upgrade batch size. -1. Click **Upgrade**. +1. Click **Save**. **Result:** The workload begins upgrading its containers, per your specifications. Note that scaling up the deployment or updating the upgrade/scaling policy won't result in the pods recreation. diff --git a/content/rancher/v2.6/en/logging/custom-resource-config/flows/_index.md b/content/rancher/v2.6/en/logging/custom-resource-config/flows/_index.md index f4ffeb05cfb..2ad4991dc8f 100644 --- a/content/rancher/v2.6/en/logging/custom-resource-config/flows/_index.md +++ b/content/rancher/v2.6/en/logging/custom-resource-config/flows/_index.md @@ -3,7 +3,7 @@ title: Flows and ClusterFlows weight: 1 --- -For the full details on configuring `Flows` and `ClusterFlows`, see the [Banzai Cloud Logging operator documentation.](https://banzaicloud.com/docs/one-eye/logging-operator/configuration/output/) +For the full details on configuring `Flows` and `ClusterFlows`, see the [Banzai Cloud Logging operator documentation.](https://banzaicloud.com/docs/one-eye/logging-operator/configuration/flow/) - [Configuration](#configuration) - [YAML Example](#yaml-example) diff --git a/content/rancher/v2.6/en/longhorn/_index.md b/content/rancher/v2.6/en/longhorn/_index.md index ad14ffbb067..fc3d2e2ac91 100644 --- a/content/rancher/v2.6/en/longhorn/_index.md +++ b/content/rancher/v2.6/en/longhorn/_index.md @@ -6,7 +6,7 @@ weight: 19 [Longhorn](https://longhorn.io/) is a lightweight, reliable and easy-to-use distributed block storage system for Kubernetes. -Longhorn is free, open source software. Originally developed by Rancher Labs, it is now being developed as a sandbox project of the Cloud Native Computing Foundation. It can be installed on any Kubernetes cluster with Helm, with kubectl, or with the Rancher UI. You can learn more about its architecture [here.](https://longhorn.io/docs/1.0.2/concepts/) +Longhorn is free, open source software. Originally developed by Rancher Labs, it is now being developed as a sandbox project of the Cloud Native Computing Foundation. It can be installed on any Kubernetes cluster with Helm, with kubectl, or with the Rancher UI. You can learn more about its architecture [here.](https://longhorn.io/docs/latest/concepts/) With Longhorn, you can: @@ -24,12 +24,12 @@ With Longhorn, you can: ### Installing Longhorn with Rancher -1. Fulfill all [Installation Requirements.](https://longhorn.io/docs/1.1.0/deploy/install/#installation-requirements) +1. Fulfill all [Installation Requirements.](https://longhorn.io/docs/latest/deploy/install/#installation-requirements) 1. Go to the cluster where you want to install Longhorn. 1. Click **Apps & Marketplace**. 1. Click **Charts**. 1. Click **Longhorn**. -1. Optional: To customize the initial settings, click **Longhorn Default Settings** and edit the configuration. For help customizing the settings, refer to the [Longhorn documentation.](https://longhorn.io/docs/1.0.2/references/settings/) +1. Optional: To customize the initial settings, click **Longhorn Default Settings** and edit the configuration. For help customizing the settings, refer to the [Longhorn documentation.](https://longhorn.io/docs/latest/references/settings/) 1. Click **Install**. **Result:** Longhorn is deployed in the Kubernetes cluster. @@ -64,7 +64,7 @@ Longhorn creates a dedicated storage controller for each volume and synchronousl The storage controller and replicas are themselves orchestrated using Kubernetes. -You can learn more about its architecture [here.](https://longhorn.io/docs/1.0.2/concepts/) +You can learn more about its architecture [here.](https://longhorn.io/docs/latest/concepts/)
Longhorn Architecture
![Longhorn Architecture]({{}}/img/rancher/longhorn-architecture.svg) diff --git a/content/rancher/v2.6/en/monitoring-alerting/configuration/_index.md b/content/rancher/v2.6/en/monitoring-alerting/configuration/_index.md index 078bf7f31e4..876031fb4d7 100644 --- a/content/rancher/v2.6/en/monitoring-alerting/configuration/_index.md +++ b/content/rancher/v2.6/en/monitoring-alerting/configuration/_index.md @@ -11,6 +11,8 @@ For information on configuring custom scrape targets and rules for Prometheus, p The resource requests and limits for the monitoring application can be configured when installing `rancher-monitoring`. For more information about the default limits, see [this page.](./helm-chart-options/#configuring-resource-limits-and-requests) +>**Note:** On an idle cluster, Monitoring V2 has significantly higher CPU usage (up to 70%) as compared to Monitoring V1. To improve performance and achieve similar results as in Monitoring V1, turn off the Prometheus adapter. + # Prometheus Configuration It is usually not necessary to directly edit the Prometheus custom resource. diff --git a/content/rancher/v2.6/en/monitoring-alerting/configuration/receiver/_index.md b/content/rancher/v2.6/en/monitoring-alerting/configuration/receiver/_index.md index d1754f86d81..3a9daaeb6ce 100644 --- a/content/rancher/v2.6/en/monitoring-alerting/configuration/receiver/_index.md +++ b/content/rancher/v2.6/en/monitoring-alerting/configuration/receiver/_index.md @@ -24,7 +24,6 @@ The [Alertmanager Config](https://prometheus.io/docs/alerting/latest/configurati - [Trusted CA for Notifiers](#trusted-ca-for-notifiers) # Creating Receivers in the Rancher UI -_Available as of v2.5.4_ > **Prerequisites:** > @@ -33,11 +32,27 @@ _Available as of v2.5.4_ To create notification receivers in the Rancher UI, -1. Go to the cluster where you want to create receivers. Click **Monitoring** and click **Receiver**. +{{% tabs %}} +{{% tab "Rancher v2.6.5+" %}} + +1. Go to the cluster where you want to create receivers. Click **Monitoring -> Alerting -> AlertManagerConfigs**. +1. Ciick **Create**. +1. Click **Add Receiver**. +1. Enter a **Name** for the receiver. +1. Configure one or more providers for the receiver. For help filling out the forms, refer to the configuration options below. +1. Click **Create**. + +{{% /tab %}} +{{% tab "Rancher before v2.6.5" %}} + +1. Go to the cluster where you want to create receivers. Click **Monitoring** and click **Receiver**. 2. Enter a name for the receiver. 3. Configure one or more providers for the receiver. For help filling out the forms, refer to the configuration options below. 4. Click **Create**. +{{% /tab %}} +{{% /tabs %}} + **Result:** Alerts can be configured to send notifications to the receiver(s). # Receiver Configuration @@ -72,7 +87,7 @@ The custom receiver option can be used to configure any receiver in YAML that ca | Field | Type | Description | |------|--------------|------| | URL | String | Enter your Slack webhook URL. For instructions to create a Slack webhook, see the [Slack documentation.](https://get.slack.help/hc/en-us/articles/115005265063-Incoming-WebHooks-for-Slack) | -| Default Channel | String | Enter the name of the channel that you want to send alert notifications in the following format: `#`. | +| Default Channel | String | Enter the name of the channel that you want to send alert notifications in the following format: `#`. | | Proxy URL | String | Proxy for the webhook notifications. | | Enable Send Resolved Alerts | Bool | Whether to send a follow-up notification if an alert has been resolved (e.g. [Resolved] High CPU Usage). | @@ -81,7 +96,7 @@ The custom receiver option can be used to configure any receiver in YAML that ca | Field | Type | Description | |------|--------------|------| | Default Recipient Address | String | The email address that will receive notifications. | -| Enable Send Resolved Alerts | Bool | Whether to send a follow-up notification if an alert has been resolved (e.g. [Resolved] High CPU Usage). | +| Enable Send Resolved Alerts | Bool | Whether to send a follow-up notification if an alert has been resolved (e.g. [Resolved] High CPU Usage). | SMTP options: @@ -100,7 +115,7 @@ SMTP options: | Integration Type | String | `Events API v2` or `Prometheus`. | | Default Integration Key | String | For instructions to get an integration key, see the [PagerDuty documentation.](https://www.pagerduty.com/docs/guides/prometheus-integration-guide/) | | Proxy URL | String | Proxy for the PagerDuty notifications. | -| Enable Send Resolved Alerts | Bool | Whether to send a follow-up notification if an alert has been resolved (e.g. [Resolved] High CPU Usage). | +| Enable Send Resolved Alerts | Bool | Whether to send a follow-up notification if an alert has been resolved (e.g. [Resolved] High CPU Usage). | # Opsgenie @@ -172,7 +187,7 @@ The SMS receiver is not a native receiver and must be enabled before it can be u 1. In the upper left corner, click **☰ > Cluster Management**. 1. On the **Clusters** page, go to the cluster where you want to install `rancher-alerting-drivers` and click **Explore**. -1. In the left navigation bar, click +1. In the left navigation bar, click 1. Click the **Alerting Drivers** app. 1. Click the **Helm Deploy Options** tab 1. Select the **SMS** option and click **Install**. @@ -224,11 +239,11 @@ You can also set up multiple receivers by using the `continue` option for a rout To set up notifications via Slack, the following Alertmanager Config YAML can be placed into the `alertmanager.yaml` key of the Alertmanager Config Secret, where the `api_url` should be updated to use your Webhook URL from Slack: ```yaml -route: +route: group_by: ['job'] group_wait: 30s group_interval: 5m - repeat_interval: 3h + repeat_interval: 3h receiver: 'slack-notifications' receivers: - name: 'slack-notifications' diff --git a/content/rancher/v2.6/en/monitoring-alerting/configuration/route/_index.md b/content/rancher/v2.6/en/monitoring-alerting/configuration/route/_index.md index 4fc0019c79f..4366f20a9a5 100644 --- a/content/rancher/v2.6/en/monitoring-alerting/configuration/route/_index.md +++ b/content/rancher/v2.6/en/monitoring-alerting/configuration/route/_index.md @@ -42,6 +42,21 @@ The route needs to refer to a [receiver](#receiver-configuration) that has alrea ### Grouping +{{% tabs %}} +{{% tab "Rancher v2.6.5+" %}} + +> **Note** As of Rancher v2.6.5 `Group By` now accepts a list of strings instead of key-value pairs. See the [upstream documentation](https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/api.md#route) for details. + +| Field | Default | Description | +|-------|--------------|---------| +| Group By | N/a | List of labels to group by. Labels must not be repeated (unique list). Special label "..." (aggregate by all possible labels), if provided, must be the only element in the list. | +| Group Wait | 30s | How long to wait to buffer alerts of the same group before sending initially. | +| Group Interval | 5m | How long to wait before sending an alert that has been added to a group of alerts for which an initial notification has already been sent. | +| Repeat Interval | 4h | How long to wait before re-sending a given alert that has already been sent. | + +{{% /tab %}} +{{% tab "Rancher before v2.6.5" %}} + | Field | Default | Description | |-------|--------------|---------| | Group By | N/a | The labels by which incoming alerts are grouped together. For example, `[ group_by: '[' , ... ']' ]` Multiple alerts coming in for labels such as `cluster=A` and `alertname=LatencyHigh` can be batched into a single group. To aggregate by all possible labels, use the special value `'...'` as the sole label name, for example: `group_by: ['...']` Grouping by `...` effectively disables aggregation entirely, passing through all alerts as-is. This is unlikely to be what you want, unless you have a very low alert volume or your upstream notification system performs its own grouping. | @@ -49,6 +64,11 @@ The route needs to refer to a [receiver](#receiver-configuration) that has alrea | Group Interval | 5m | How long to wait before sending an alert that has been added to a group of alerts for which an initial notification has already been sent. | | Repeat Interval | 4h | How long to wait before re-sending a given alert that has already been sent. | +{{% /tab %}} +{{% /tabs %}} + + + ### Matching The **Match** field refers to a set of equality matchers used to identify which alerts to send to a given Route based on labels defined on that alert. When you add key-value pairs to the Rancher UI, they correspond to the YAML in this format: diff --git a/content/rancher/v2.6/en/monitoring-alerting/configuration/servicemonitor-podmonitor/_index.md b/content/rancher/v2.6/en/monitoring-alerting/configuration/servicemonitor-podmonitor/_index.md index 20dc901da31..d2f848b6e67 100644 --- a/content/rancher/v2.6/en/monitoring-alerting/configuration/servicemonitor-podmonitor/_index.md +++ b/content/rancher/v2.6/en/monitoring-alerting/configuration/servicemonitor-podmonitor/_index.md @@ -26,6 +26,6 @@ For more information about how ServiceMonitors work, refer to the [Prometheus Op This pseudo-CRD maps to a section of the Prometheus custom resource configuration. It declaratively specifies how group of pods should be monitored. -When a PodMonitor is created, the Prometheus Operator updates the Prometheus scrape configuration to include the PodMonitor configuration. Then Prometheus begins scraping metrics from the endpoint defined in the ServiceMonitor. +When a PodMonitor is created, the Prometheus Operator updates the Prometheus scrape configuration to include the PodMonitor configuration. Then Prometheus begins scraping metrics from the endpoint defined in the PodMonitor. Any Pods in your cluster that match the labels located within the PodMonitor `selector` field will be monitored based on the `podMetricsEndpoints` specified on the PodMonitor. For more information on what fields can be specified, please look at the [spec](https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/api.md#podmonitorspec) provided by Prometheus Operator. diff --git a/content/rancher/v2.6/en/monitoring-alerting/how-monitoring-works/_index.md b/content/rancher/v2.6/en/monitoring-alerting/how-monitoring-works/_index.md index 9e7690d7de3..30b19cdbc66 100644 --- a/content/rancher/v2.6/en/monitoring-alerting/how-monitoring-works/_index.md +++ b/content/rancher/v2.6/en/monitoring-alerting/how-monitoring-works/_index.md @@ -8,56 +8,81 @@ weight: 1 3. [How Alertmanager Works](#3-how-alertmanager-works) 4. [Monitoring V2 Specific Components](#4-monitoring-v2-specific-components) 5. [Scraping and Exposing Metrics](#5-scraping-and-exposing-metrics) -6. [Monitoring on RKE2 Clusters](#6-monitoring-on-rke2-clusters) # 1. Architecture Overview -This diagram shows how data flows through the Monitoring V2 application: +_**The following sections describe how data flows through the Monitoring V2 application:**_ -{{% row %}} -{{% column %}} +### Prometheus Operator -![How data flows through the monitoring application]({{}}/img/rancher/monitoring-v2-architecture-overview.svg) +Prometheus Operator observes ServiceMonitors, PodMonitors, and PrometheusRules being created. When the Prometheus configuration resources are created, Prometheus Operator calls the Prometheus API to sync the new configuration. As the diagram at the end of this section shows, the Prometheus Operator acts as the intermediary between Prometheus and Kubernetes, calling the Prometheus API to synchronize Prometheus with the monitoring-related resources in Kubernetes. -{{% /column %}} -{{% column %}} +### ServiceMonitors and PodMonitors +ServiceMonitors and PodMonitors declaratively specify targets, such as Services and Pods, that need to be monitored. -1. Rules define what Prometheus metrics or time series database queries should result in alerts being fired. -2. ServiceMonitors and PodMonitors declaratively specify how services and pods should be monitored. They use labels to scrape metrics from pods. -3. Prometheus Operator observes ServiceMonitors, PodMonitors and PrometheusRules being created. -4. When the Prometheus configuration resources are created, Prometheus Operator calls the Prometheus API to sync the new configuration. -5. Recording Rules are not directly used for alerting. They create new time series of precomputed queries. These new time series data can then be queried to generate alerts. -6. Prometheus scrapes all targets in the scrape configuration on a recurring schedule based on the scrape interval, storing the results in its time series database.Depending on the Kubernetes master component and Kubernetes distribution, the metrics from a certain Kubernetes component could be directly exposed to Prometheus, proxied through PushProx, or not available. For details, see Scraping and Exposing Metrics. -7. Prometheus evaluates the alerting rules against the time series database. It fires alerts to Alertmanager whenever an alerting rule evaluates to a positive number. -8. Alertmanager uses routes to group, label and filter the fired alerts to translate them into useful notifications. -9. Alertmanager uses the Receiver configuration to send notifications to Slack, PagerDuty, SMS, or other types of receivers. +- Targets are scraped on a recurring schedule based on the configured Prometheus scrape interval, and the metrics that are scraped are stored into the Prometheus Time Series Database (TSDB). -{{% /column %}} -{{% /row %}} +- In order to perform the scrape, ServiceMonitors and PodMonitors are defined with label selectors that determine which Services or Pods should be scraped and endpoints that determine how the scrape should happen on the given target, e.g., scrape/metrics in TCP 10252, proxying through IP addr x.x.x.x. +- Out of the box, Monitoring V2 comes with certain pre-configured exporters that are deployed based on the type of Kubernetes cluster that it is deployed on. For more information, see [Scraping and Exposing Metrics](#5-scraping-and-exposing-metrics). +### How PushProx Works + +- Certain internal Kubernetes components are scraped via a proxy deployed as part of Monitoring V2 called **PushProx**. The Kubernetes components that expose metrics to Prometheus through PushProx are the following: +`kube-controller-manager`, `kube-scheduler`, `etcd`, and `kube-proxy`. + +- For each PushProx exporter, we deploy one PushProx client onto all target nodes. For example, a PushProx client is deployed onto all controlplane nodes for kube-controller-manager, all etcd nodes for kube-etcd, and all nodes for kubelet. + +- We deploy exactly one PushProx proxy per exporter. The process for exporting metrics is as follows: + +1. The PushProx Client establishes an outbound connection with the PushProx Proxy. +1. The client then polls the proxy for scrape requests that have come into the proxy. +1. When the proxy receives a scrape request from Prometheus, the client sees it as a result of the poll. +1. The client scrapes the internal component. +1. The internal component responds by pushing metrics back to the proxy. + + +

Process for Exporting Metrics with PushProx:
+ +![Process for Exporting Metrics with PushProx]({{}}/img/rancher/pushprox-process.svg) + +### PrometheusRules + +PrometheusRules allow users to define rules for what metrics or time series database queries should result in alerts being fired. Rules are evaluated on an interval. + +- **Recording rules** create a new time series based on existing series that have been collected. They are frequently used to precompute complex queries. +- **Alerting rules** run a particular query and fire an alert from Prometheus if the query evaluates to a non-zero value. + +### Alert Routing + +Once Prometheus determines that an alert needs to be fired, alerts are forwarded to **Alertmanager**. + +- Alerts contain labels that come from the PromQL query itself and additional labels and annotations that can be provided as part of specifying the initial PrometheusRule. + +- Before receiving any alerts, Alertmanager will use the **routes** and **receivers** specified in its configuration to form a routing tree on which all incoming alerts are evaluated. Each node of the routing tree can specify additional grouping, labeling, and filtering that needs to happen based on the labels attached to the Prometheus alert. A node on the routing tree (usually a leaf node) can also specify that an alert that reaches it needs to be sent out to a configured Receiver, e.g., Slack, PagerDuty, SMS, etc. Note that Alertmanager will send an alert first to **alertingDriver**, then alertingDriver will send or forward alert to the proper destination. + +- Routes and receivers are also stored in the Kubernetes API via the Alertmanager Secret. When the Secret is updated, Alertmanager is also updated automatically. Note that routing occurs via labels only (not via annotations, etc.). + +
How data flows through the monitoring application:
# 2. How Prometheus Works -### 2.1. Storing Time Series Data +### Storing Time Series Data After collecting metrics from exporters, Prometheus stores the time series in a local on-disk time series database. Prometheus optionally integrates with remote systems, but `rancher-monitoring` uses local storage for the time series database. -The database can then be queried using PromQL, the query language for Prometheus. Grafana dashboards use PromQL queries to generate data visualizations. +Once stored, users can query this TSDB using PromQL, the query language for Prometheus. -### 2.2. Querying the Time Series Database +PromQL queries can be visualized in one of two ways: -The PromQL query language is the primary tool to query Prometheus for time series data. +1. By supplying the query in Prometheus's Graph UI, which will show a simple graphical view of the data. +1. By creating a Grafana Dashboard that contains the PromQL query and additional formatting directives that label axes, add units, change colors, use alternative visualizations, etc. -In Grafana, you can right-click a CPU utilization and click Inspect. This opens a panel that shows the [raw query results.](https://grafana.com/docs/grafana/latest/panels/inspect-panel/#inspect-raw-query-results)The raw results demonstrate how each dashboard is powered by PromQL queries. +### Defining Rules for Prometheus -### 2.3. Defining Rules for when Alerts Should be Fired - -Rules define the conditions for Prometheus to fire alerts. When PrometheusRule custom resources are created or updated, the Prometheus Operator observes the change and calls the Prometheus API to synchronize the rule configuration with the Alerting Rules and Recording Rules in Prometheus. - -When you define a Rule (which is declared within a RuleGroup in a PrometheusRule resource), the [spec of the Rule itself](https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/api.md#rule) contains labels that are used by Alertmanager to figure out which Route should receive this Alert. For example, an Alert with the label `team: front-end` will be sent to all Routes that match on that label. +Rules define queries that Prometheus needs to execute on a regular `evaluationInterval` to perform certain actions, such as firing an alert (alerting rules) or precomputing a query based on others existing in its TSDB (recording rules). These rules are encoded in PrometheusRules custom resources. When PrometheusRule custom resources are created or updated, the Prometheus Operator observes the change and calls the Prometheus API to synchronize the set of rules that Prometheus is currently evaluating on a regular interval. A PrometheusRule allows you to define one or more RuleGroups. Each RuleGroup consists of a set of Rule objects that can each represent either an alerting or a recording rule with the following fields: @@ -66,7 +91,9 @@ A PrometheusRule allows you to define one or more RuleGroups. Each RuleGroup con - Labels that should be attached to the alert or record that identify it (e.g. cluster name or severity) - Annotations that encode any additional important pieces of information that need to be displayed on the notification for an alert (e.g. summary, description, message, runbook URL, etc.). This field is not required for recording rules. -### 2.4. Firing Alerts +On evaluating a [rule](https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/api.md#rule), Prometheus will execute the provided PromQL query, add additional provided labels (or annotations - only for alerting rules), and execute the appropriate action for the rule. For example, an Alerting Rule that adds `team: front-end` as a label to the provided PromQL query will append that label to the fired alert, which will allow Alertmanager to forward the alert to the correct Receiver. + +### Alerting and Recording Rules Prometheus doesn't maintain the state of whether alerts are active. It fires alerts repetitively at every evaluation interval, relying on Alertmanager to group and filter the alerts into meaningful notifications. @@ -81,6 +108,7 @@ Alerting rules are more commonly used. Whenever an alerting rule evaluates to a The Rule file adds labels and annotations to alerts before firing them, depending on the use case: - Labels indicate information that identifies the alert and could affect the routing of the alert. For example, if when sending an alert about a certain container, the container ID could be used as a label. + - Annotations denote information that doesn't affect where an alert is routed, for example, a runbook or an error message. # 3. How Alertmanager Works @@ -88,17 +116,24 @@ The Rule file adds labels and annotations to alerts before firing them, dependin The Alertmanager handles alerts sent by client applications such as the Prometheus server. It takes care of the following tasks: - Deduplicating, grouping, and routing alerts to the correct receiver integration such as email, PagerDuty, or OpsGenie + - Silencing and inhibition of alerts + - Tracking alerts that fire over time + - Sending out the status of whether an alert is currently firing, or if it is resolved + +### Alerts Forwarded by alertingDrivers + +When alertingDrivers are installed, this creates a `Service` that can be used as the receiver's URL for Teams or SMS, based on the alertingDriver's configuration. The URL in the Receiver points to the alertingDrivers; so the Alertmanager sends alert first to alertingDriver, then alertingDriver forwards or sends alert to the proper destination. -### 3.1. Routing Alerts to Receivers +### Routing Alerts to Receivers Alertmanager coordinates where alerts are sent. It allows you to group alerts based on labels and fire them based on whether certain labels are matched. One top-level route accepts all alerts. From there, Alertmanager continues routing alerts to receivers based on whether they match the conditions of the next route. -While the Rancher UI forms only allow editing a routing tree that is two levels deep, you can configure more deeply nested routing structures by editing the Alertmanager custom resource YAML. +While the Rancher UI forms only allow editing a routing tree that is two levels deep, you can configure more deeply nested routing structures by editing the Alertmanager Secret. -### 3.2. Configuring Multiple Receivers +### Configuring Multiple Receivers By editing the forms in the Rancher UI, you can set up a Receiver resource with all the information Alertmanager needs to send alerts to your notification system. @@ -110,120 +145,89 @@ Prometheus Operator introduces a set of [Custom Resource Definitions](https://gi Prometheus Operator will automatically update your Prometheus configuration based on the live state of the resources and configuration options that are edited in the Rancher UI. -### 4.1. Resources Deployed by Default +### Resources Deployed by Default By default, a set of resources curated by the [kube-prometheus](https://github.com/prometheus-operator/kube-prometheus) project are deployed onto your cluster as part of installing the Rancher Monitoring Application to set up a basic Monitoring/Alerting stack. The resources that get deployed onto your cluster to support this solution can be found in the [`rancher-monitoring`](https://github.com/rancher/charts/tree/main/charts/rancher-monitoring) Helm chart, which closely tracks the upstream [kube-prometheus-stack](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack) Helm chart maintained by the Prometheus community with certain changes tracked in the [CHANGELOG.md](https://github.com/rancher/charts/blob/main/charts/rancher-monitoring/CHANGELOG.md). -There are also certain special types of ConfigMaps and Secrets such as those corresponding to Grafana Dashboards, Grafana Datasources, and Alertmanager Configs that will automatically update your Prometheus configuration via sidecar proxies that observe the live state of those resources within your cluster. +### Default Exporters -### 4.2. PushProx +Monitoring V2 deploys three default exporters that provide additional metrics for Prometheus to store: -PushProx enhances the security of the monitoring application, allowing it to be installed on hardened Kubernetes clusters. +1. `node-exporter`: exposes hardware and OS metrics for Linux hosts. For more information on `node-exporter`, refer to the [upstream documentation](https://prometheus.io/docs/guides/node-exporter/). -To expose Kubernetes metrics, PushProxes use a client proxy model to expose specific ports within default Kubernetes components. Node exporters expose metrics to PushProx through an outbound connection. +1. `windows-exporter`: exposes hardware and OS metrics for Windows hosts (only deployed on Windows clusters). For more information on `windows-exporter`, refer to the [upstream documentation](https://github.com/prometheus-community/windows_exporter). -The proxy allows `rancher-monitoring` to scrape metrics from processes on the hostNetwork, such as the `kube-api-server`, without opening up node ports to inbound connections. +1. `kube-state-metrics`: expose additional metrics that track the state of resources contained in the Kubernetes API (e.g., pods, workloads, etc.). For more information on `kube-state-metrics`, refer to the [upstream documentation](https://github.com/kubernetes/kube-state-metrics/tree/master/docs). -PushProx is a DaemonSet that listens for clients that seek to register. Once registered, it proxies scrape requests through the established connection. Then the client executes the request to etcd. +ServiceMonitors and PodMonitors will scrape these exporters, as defined [here](#defining-what-metrics-are-scraped). Prometheus stores these metrics, and you can query the results via either Prometheus's UI or Grafana. -All of the default ServiceMonitors, such as `rancher-monitoring-kube-controller-manager`, are configured to hit the metrics endpoint of the client using this proxy. +See the [architecture](#1-architecture-overview) section for more information on recording rules, alerting rules, and Alertmanager. -For more details about how PushProx works, refer to [Scraping Metrics with PushProx.](#5-5-scraping-metrics-with-pushprox) - - -### 4.3. Default Exporters - -`rancher-monitoring` deploys two exporters to expose metrics to prometheus: `node-exporter` and `windows-exporter`. Both are deployed as DaemonSets. - -`node-exporter` exports container, pod and node metrics for CPU and memory from each Linux node. `windows-exporter` does the same, but for Windows nodes. - -For more information on `node-exporter`, refer to the [upstream documentation.](https://prometheus.io/docs/guides/node-exporter/) - -[kube-state-metrics](https://github.com/kubernetes/kube-state-metrics) is also useful because it exports metrics for Kubernetes components. - -# 4.4. Components Exposed in the Rancher UI +### Components Exposed in the Rancher UI When the monitoring application is installed, you will be able to edit the following components in the Rancher UI: | Component | Type of Component | Purpose and Common Use Cases for Editing | |--------------|------------------------|---------------------------| -| ServiceMonitor | Custom resource | Set up targets to scrape custom metrics from. Automatically updates the scrape configuration in the Prometheus custom resource. | -| PodMonitor | Custom resource | Set up targets to scrape custom metrics from. Automatically updates the scrape configuration in the Prometheus custom resource. | -| Receiver | Configuration block (part of Alertmanager) | Set up a notification system to receive alerts. Automatically updates the Alertmanager custom resource. | -| Route | Configuration block (part of Alertmanager) | Add identifying information to make alerts more meaningful and direct them to individual teams. Automatically updates the Alertmanager custom resource. | -| PrometheusRule | Custom resource | For more advanced use cases, you may want to define what Prometheus metrics or time series database queries should result in alerts being fired. Automatically updates the Prometheus custom resource. | -| Alertmanager | Custom resource | Edit this custom resource only if you need more advanced configuration options beyond what the Rancher UI exposes in the Routes and Receivers sections. For example, you might want to edit this resource to add a routing tree with more than two levels. | -| Prometheus | Custom resource | Edit this custom resource only if you need more advanced configuration beyond what can be configured using ServiceMonitors, PodMonitors, or [Rancher monitoring Helm chart options.](../configuration/helm-chart-options) | +| ServiceMonitor | Custom resource | Sets up Kubernetes Services to scrape custom metrics from. Automatically updates the scrape configuration in the Prometheus custom resource. | +| PodMonitor | Custom resource | Sets up Kubernetes Pods to scrape custom metrics from. Automatically updates the scrape configuration in the Prometheus custom resource. | +| Receiver | Configuration block (part of Alertmanager) | Modifies information on where to send an alert (e.g., Slack, PagerDuty, etc.) and any necessary information to send the alert (e.g., TLS certs, proxy URLs, etc.). Automatically updates the Alertmanager custom resource. | +| Route | Configuration block (part of Alertmanager) | Modifies the routing tree that is used to filter, label, and group alerts based on labels and send them to the appropriate Receiver. Automatically updates the Alertmanager custom resource. | +| PrometheusRule | Custom resource | Defines additional queries that need to trigger alerts or define materialized views of existing series that are within Prometheus's TSDB. Automatically updates the Prometheus custom resource. | + +### PushProx + +PushProx allows Prometheus to scrape metrics across a network boundary, which prevents users from having to expose metrics ports for internal Kubernetes components on each node in a Kubernetes cluster. + +Since the metrics for Kubernetes components are generally exposed on the host network of nodes in the cluster, PushProx deploys a DaemonSet of clients that sit on the hostNetwork of each node and make an outbound connection to a single proxy that is sitting on the Kubernetes API. Prometheus can then be configured to proxy scrape requests through the proxy to each client, which allows it to scrape metrics from the internal Kubernetes components without requiring any inbound node ports to be open. + +Refer to [Scraping Metrics with PushProx](#scraping-metrics-with-pushprox) for more. # 5. Scraping and Exposing Metrics -### 5.1. Defining what Metrics are Scraped +### Defining what Metrics are Scraped -ServiceMonitors define targets that are intended for Prometheus to scrape. The [Prometheus custom resource tells](https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/design.md#prometheus) Prometheus which ServiceMonitors it should use to find out where to scrape metrics from. +ServiceMonitors and PodMonitors define targets that are intended for Prometheus to scrape. The [Prometheus custom resource](https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/design.md#prometheus) tells Prometheus which ServiceMonitors or PodMonitors it should use to find out where to scrape metrics from. -The Prometheus Operator observes the ServiceMonitors. When it observes that ServiceMonitors are created or updated, it calls the Prometheus API to update the scrape configuration in the Prometheus custom resource and keep it in sync with the scrape configuration in the ServiceMonitors. This scrape configuration tells Prometheus which endpoints to scrape metrics from and how it will label the metrics from those endpoints. +The Prometheus Operator observes the ServiceMonitors and PodMonitors. When it observes that they are created or updated, it calls the Prometheus API to update the scrape configuration in the Prometheus custom resource and keep it in sync with the scrape configuration in the ServiceMonitors or PodMonitors. This scrape configuration tells Prometheus which endpoints to scrape metrics from and how it will label the metrics from those endpoints. Prometheus scrapes all of the metrics defined in its scrape configuration at every `scrape_interval`, which is one minute by default. The scrape configuration can be viewed as part of the Prometheus custom resource that is exposed in the Rancher UI. -### 5.2. How the Prometheus Operator Sets up Metrics Scraping +### How the Prometheus Operator Sets up Metrics Scraping The Prometheus Deployment or StatefulSet scrapes metrics, and the configuration of Prometheus is controlled by the Prometheus custom resources. The Prometheus Operator watches for Prometheus and Alertmanager resources, and when they are created, the Prometheus Operator creates a Deployment or StatefulSet for Prometheus or Alertmanager with the user-defined configuration. -
How the Prometheus Operator Sets up Metrics Scraping
+When the Prometheus Operator observes ServiceMonitors, PodMonitors, and PrometheusRules being created, it knows that the scrape configuration needs to be updated in Prometheus. It updates Prometheus by first updating the configuration and rules files in the volumes of Prometheus's Deployment or StatefulSet. Then it calls the Prometheus API to sync the new configuration, resulting in the Prometheus Deployment or StatefulSet to be modified in place. -![How the Prometheus Operator sets up metrics scraping]({{}}/img/rancher/set-up-scraping.svg) - -When the Prometheus Operator observes ServiceMonitors, PodMonitors and PrometheusRules being created, it knows that the scrape configuration needs to be updated in Prometheus. It updates Prometheus by first updating the configuration and rules files in the volumes of Prometheus's Deployment or StatefulSet. Then it calls the Prometheus API to sync the new configuration, resulting in the Prometheus Deployment or StatefulSet to be modified in place. - -![How the Prometheus Operator Updates Scrape Configuration]({{}}/img/rancher/update-scrape-config.svg) - -### 5.3. How Kubernetes Component Metrics are Exposed +### How Kubernetes Component Metrics are Exposed Prometheus scrapes metrics from deployments known as [exporters,](https://prometheus.io/docs/instrumenting/exporters/) which export the time series data in a format that Prometheus can ingest. In Prometheus, time series consist of streams of timestamped values belonging to the same metric and the same set of labeled dimensions. -To allow monitoring to be installed on hardened Kubernetes clusters, `rancher-monitoring` application proxies the communication between Prometheus and the exporter through PushProx for some Kubernetes master components. +### Scraping Metrics with PushProx -### 5.4. Scraping Metrics without PushProx +Certain internal Kubernetes components are scraped via a proxy deployed as part of Monitoring V2 called PushProx. For detailed information on PushProx, refer [here](#how-pushprox-works) and to the above [architecture](#1-architecture-overview) section. -The Kubernetes components that directly expose metrics to Prometheus are the following: +### Scraping Metrics -- kubelet -- ingress-nginx* +The following Kubernetes components are directly scraped by Prometheus: + +- kubelet* +- ingress-nginx** - coreDns/kubeDns - kube-api-server -\* For RKE and RKE2 clusters, ingress-nginx is deployed by default and treated as an internal Kubernetes component. +\* You can optionally use `hardenedKubelet.enabled` to use a PushProx, but that is not the default. -### 5.5. Scraping Metrics with PushProx +** For RKE and RKE2 clusters, ingress-nginx is deployed by default and treated as an internal Kubernetes component. -The purpose of this architecture is to allow us to scrape internal Kubernetes components without exposing those ports to inbound requests. As a result, Prometheus can scrape metrics across a network boundary. -The Kubernetes components that expose metrics to Prometheus through PushProx are the following: +### Scraping Metrics Based on Kubernetes Distribution -- kube-controller-manager -- kube-scheduler -- etcd -- kube-proxy - -For each PushProx exporter, we deploy one PushProx client onto all target nodes. For example, a PushProx client is deployed onto all controlplane nodes for kube-controller-manager, all etcd nodes for kube-etcd, and all nodes for kubelet. We deploy exactly one PushProx proxy per exporter. - -The process for exporting metrics is as follows: - -1. The PushProx Client establishes an outbound connection with the PushProx Proxy. -2. The client then polls the proxy for scrape requests that have come into the proxy. -3. When the proxy receives a scrape request from Prometheus, the client sees it as a result of the poll. -4. The client scrapes the internal component. -5. The internal component responds by pushing metrics back to the proxy. - -
Process for Exporting Metrics with PushProx
- -![Process for Exporting Metrics with PushProx]({{}}/img/rancher/pushprox-process.svg) - -Metrics are scraped differently based on the Kubernetes distribution. For help with terminology, see Terminology(#terminology). For details, see the table below: +Metrics are scraped differently based on the Kubernetes distribution. For help with terminology, refer [here](#terminology). For details, see the table below:
How Metrics are Exposed to Prometheus
@@ -240,7 +244,7 @@ Metrics are scraped differently based on the Kubernetes distribution. For help w \* For RKE and RKE2 clusters, ingress-nginx is deployed by default and treated as an internal Kubernetes component. -### 5.6. Terminology +### Terminology - **kube-scheduler:** The internal Kubernetes component that uses information in the pod spec to decide on which node to run a pod. - **kube-controller-manager:** The internal Kubernetes component that is responsible for node management (detecting if a node fails), pod replication and endpoint creation. @@ -250,13 +254,3 @@ Metrics are scraped differently based on the Kubernetes distribution. For help w - **ingress-nginx:** An Ingress controller for Kubernetes using NGINX as a reverse proxy and load balancer. - **coreDns/kubeDns:** The internal Kubernetes component responsible for DNS. - **kube-api-server:** The main internal Kubernetes component that is responsible for exposing APIs for the other master components. - -# 6. Monitoring on RKE2 Clusters - -Rancher v2.6 introduced the ability to provision new Kubernetes clusters with [RKE2,](https://docs.rke2.io/) which is Rancher's fully conformant Kubernetes distribution that focuses on security and compliance within the U.S. Federal Government sector. To allow Monitoring V2 to be installed on RKE2 Kubernetes clusters, the `rkeIngressNginx` and `rke2IngressNginx` sub-charts were introduced to scrape metrics from the `ingress-nginx` Deployment/DaemonSet in RKE and RKE2 clusters respectively. - -The PushProx pod needs to run on the same nodes as the `ingress-nginx` pod. - -When the RKE2 cluster's Kubernetes version is <= 1.20, the workload type of `ingress-nginx` is a Deployment. The `pushprox-ingress-nginx-client` is deployed as a Deployment, and the Rancher UI sets the Helm chart value `rke2IngressNginx.deployment.enabled=true`. - -For Kubernetes >= 1.21, the workload type of `ingress-nginx` is a DaemonSet. The `pushprox-ingress-nginx-client` is deployed as a DaemonSet, which is the default behavior. \ No newline at end of file diff --git a/content/rancher/v2.6/en/monitoring-alerting/prometheus-federator/_index.md b/content/rancher/v2.6/en/monitoring-alerting/prometheus-federator/_index.md new file mode 100644 index 00000000000..cc88321bee4 --- /dev/null +++ b/content/rancher/v2.6/en/monitoring-alerting/prometheus-federator/_index.md @@ -0,0 +1,83 @@ +--- +title: Prometheus Federator +weight: 7 +--- + +Prometheus Federator, also referred to as Project Monitoring v2, deploys a Helm Project Operator (based on the [rancher/helm-project-operator](https://github.com/rancher/helm-project-operator)), an operator that manages deploying Helm charts each containing a Project Monitoring Stack, where each stack contains: + +- [Prometheus](https://prometheus.io/) (managed externally by [Prometheus Operator](https://github.com/prometheus-operator/prometheus-operator)) +- [Alertmanager](https://prometheus.io/docs/alerting/latest/alertmanager/) (managed externally by [Prometheus Operator](https://github.com/prometheus-operator/prometheus-operator)) +- [Grafana](https://github.com/helm/charts/tree/master/stable/grafana) (deployed via an embedded Helm chart) +- Default PrometheusRules and Grafana dashboards based on the collection of community-curated resources from [kube-prometheus](https://github.com/prometheus-operator/kube-prometheus/) +- Default ServiceMonitors that watch the deployed resources + +> **Important** Prometheus Federator is designed to be deployed alongside an existing Prometheus Operator deployment in a cluster that has already installed the Prometheus Operator CRDs. + +## How does the operator work? + +1. On deploying this chart, users can create ProjectHelmCharts CRs with `spec.helmApiVersion` set to `monitoring.cattle.io/v1alpha1` (also known as "Project Monitors" in the Rancher UI) in a **Project Registration Namespace (`cattle-project-`)**. +2. On seeing each ProjectHelmChartCR, the operator will automatically deploy a Project Prometheus stack on the Project Owner's behalf in the **Project Release Namespace (`cattle-project--monitoring`)** based on a HelmChart CR and a HelmRelease CR automatically created by the ProjectHelmChart controller in the **Operator / System Namespace**. +3. RBAC will automatically be assigned in the Project Release Namespace to allow users to view the Prometheus, Alertmanager, and Grafana UIs of the Project Monitoring Stack deployed; this will be based on RBAC defined on the Project Registration Namespace against the [default Kubernetes user-facing roles](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#user-facing-roles). For more information, see the section on [configuring RBAC](./rbac/). + +### What is a Project? + +In Prometheus Federator, a Project is a group of namespaces that can be identified by a `metav1.LabelSelector`. By default, the label used to identify projects is `field.cattle.io/projectId`, the label used to identify namespaces that are contained within a given Rancher Project. + +### Configuring the Helm release created by a ProjectHelmChart + +The `spec.values` of this ProjectHelmChart's resources will correspond to the `values.yaml` override to be supplied to the underlying Helm chart deployed by the operator on the user's behalf; to see the underlying chart's `values.yaml` spec, either: + +- View the chart's definition located at [`rancher/prometheus-federator` under `charts/rancher-project-monitoring`](https://github.com/rancher/prometheus-federator/blob/main/charts/rancher-project-monitoring) (where the chart version will be tied to the version of this operator). +- Look for the ConfigMap named `monitoring.cattle.io.v1alpha1` that is automatically created in each Project Registration Namespace, which will contain both the `values.yaml` and `questions.yaml` that was used to configure the chart (which was embedded directly into the `prometheus-federator` binary). + +### Namespaces + +As a Project Operator based on [rancher/helm-project-operator](https://github.com/rancher/helm-project-operator), Prometheus Federator has three different classifications of namespaces that the operator looks out for: + +1. **Operator / System Namespace**: The namespace that the operator is deployed into (e.g., `cattle-monitoring-system`). This namespace will contain all HelmCharts and HelmReleases for all ProjectHelmCharts watched by this operator. **Only Cluster Admins should have access to this namespace.** +2. **Project Registration Namespace (`cattle-project-`)**: The set of namespaces that the operator watches for ProjectHelmCharts within. The RoleBindings and ClusterRoleBindings that apply to this namespace will also be the source of truth for the auto-assigned RBAC created in the Project Release Namespace. For details, refer to the [RBAC page](./rbac/). **Project Owners (admin), Project Members (edit), and Read-Only Members (view) should have access to this namespace.** + + > **Note:** Project Registration Namespaces will be auto-generated by the operator and imported into the Project it is tied to if `.Values.global.cattle.projectLabel` is provided, which is set to `field.cattle.io/projectId` by default. This indicates that a Project Registration Namespace should be created by the operator if at least one namespace is observed with that label. The operator will not let these namespaces be deleted unless either all namespaces with that label are gone (e.g., this is the last namespace in that project, in which case the namespace will be marked with the label `"helm.cattle.io/helm-project-operator-orphaned": "true"`, which signals that it can be deleted), or it is no longer watching that project because the project ID was provided under `.Values.helmProjectOperator.otherSystemProjectLabelValues`, which serves as a denylist for Projects. These namespaces will also never be auto-deleted to avoid destroying user data; it is recommended that users clean up these namespaces manually if desired on creating or deleting a project. + + > **Note:** If `.Values.global.cattle.projectLabel` is not provided, the Operator / System Namespace will also be the Project Registration Namespace. +3. **Project Release Namespace (`cattle-project--monitoring`):** The set of namespaces that the operator deploys Project Monitoring Stacks within on behalf of a ProjectHelmChart; the operator will also automatically assign RBAC to Roles created in this namespace by the Project Monitoring Stack based on bindings found in the Project Registration Namespace. **Only Cluster Admins should have access to this namespace; Project Owners (admin), Project Members (edit), and Read-Only Members (view) will be assigned limited access to this namespace by the deployed Helm Chart and Prometheus Federator.** + + > **Note:** Project Release Namespaces are automatically deployed and imported into the project whose ID is specified under `.Values.helmProjectOperator.projectReleaseNamespaces.labelValue`, which defaults to the value of `.Values.global.cattle.systemProjectId` if not specified, whenever a ProjectHelmChart is specified in a Project Registration Namespace. + + > **Note:** Project Release Namespaces follow the same orphaning conventions as Project Registration Namespaces (see note above). + + > **Note:** If `.Values.projectReleaseNamespaces.enabled` is false, the Project Release Namespace will be the same as the Project Registration Namespace. + +### Helm Resources (HelmChart, HelmRelease) + +On deploying a ProjectHelmChart, the Prometheus Federator will automatically create and manage two child custom resources that manage the underlying Helm resources in turn: + +- A HelmChart CR (managed via an embedded [k3s-io/helm-contoller](https://github.com/k3s-io/helm-controller) in the operator): This custom resource automatically creates a Job in the same namespace that triggers a `helm install`, `helm upgrade`, or `helm uninstall` depending on the change applied to the HelmChart CR. This CR is automatically updated on changes to the ProjectHelmChart (e.g., modifying the values.yaml) or changes to the underlying Project definition (e.g., adding or removing namespaces from a project). + +> **Important:** If a ProjectHelmChart is not deploying or updating the underlying Project Monitoring Stack for some reason, the Job created by this resource in the Operator / System namespace should be the first place you check to see if there's something wrong with the Helm operation. However, this is generally only accessible by a **Cluster Admin.** + +- A HelmRelease CR (managed via an embedded [rancher/helm-locker](https://github.com/rancher/helm-locker) in the operator): This custom resource automatically locks a deployed Helm release in place and automatically overwrites updates to underlying resources unless the change happens via a Helm operation (`helm install`, `helm upgrade`, or `helm uninstall` performed by the HelmChart CR). + +> **Note:** HelmRelease CRs emit Kubernetes Events that detect when an underlying Helm release is being modified and locks it back to place. To view these events, you can use `kubectl describe helmrelease -n `; you can also view the logs on this operator to see when changes are detected and which resources modifications were attempted on. + +Both of these resources are created for all Helm charts in the Operator / System namespaces to avoid escalation of privileges to underprivileged users. + +### Advanced Helm Project Operator Configuration + +For more information on advanced configurations, refer to [this page](https://github.com/rancher/prometheus-federator/blob/main/charts/prometheus-federator/0.0.1/README.md#advanced-helm-project-operator-configuration). + + + +### Prometheus Federator on the Local Cluster + +Prometheus Federator is a resource intensive application. Installing it to the local cluster is possible, but **not recommended**. \ No newline at end of file diff --git a/content/rancher/v2.6/en/monitoring-alerting/prometheus-federator/guides/customizing-grafana/_index.md b/content/rancher/v2.6/en/monitoring-alerting/prometheus-federator/guides/customizing-grafana/_index.md new file mode 100644 index 00000000000..f5cc1e5653d --- /dev/null +++ b/content/rancher/v2.6/en/monitoring-alerting/prometheus-federator/guides/customizing-grafana/_index.md @@ -0,0 +1,8 @@ +--- +title: Customizing Grafana Dashboards +weight: 3 +--- + +Grafana dashboards are customized the same way whether it's for rancher-monitoring or for Prometheus Federator. + +For instructions, refer to [this page](../../../guides/customize-grafana/). \ No newline at end of file diff --git a/content/rancher/v2.6/en/monitoring-alerting/prometheus-federator/guides/enable-prom-fed/_index.md b/content/rancher/v2.6/en/monitoring-alerting/prometheus-federator/guides/enable-prom-fed/_index.md new file mode 100644 index 00000000000..ae4ff801de2 --- /dev/null +++ b/content/rancher/v2.6/en/monitoring-alerting/prometheus-federator/guides/enable-prom-fed/_index.md @@ -0,0 +1,62 @@ +--- +title: Enable Prometheus Federator +weight: 1 +--- + +- [Requirements](#requirements) +- [Install the Prometheus Federator Application](#install-the-prometheus-federator-application) + +# Requirements + +By default, Prometheus Federator is configured and intended to be deployed alongside [rancher-monitoring](https://rancher.com/docs/rancher/v2.6/en/monitoring-alerting/), which deploys Prometheus Operator alongside a Cluster Prometheus that each Project Monitoring Stack is configured to federate namespace-scoped metrics from by default. + +For instructions on installing rancher-monitoring, refer to [this page](../../../guides/enable-monitoring/). + +The default configuration should already be compatible with your rancher-monitoring stack. However, to optimize the security and usability of Prometheus Federator in your cluster, we recommend making these additional configurations to rancher-monitoring: + +- [Ensure the cattle-monitoring-system namespace is placed into the System Project](#ensure-the-cattle-monitoring-system-namespace-is-placed-into-the-system-project-or-a-similarly-locked-down-project-that-has-access-to-other-projects-in-the-cluster). +- [Configure rancher-monitoring to only watch for resources created by the Helm chart itself](#configure-rancher-monitoring-to-only-watch-for-resources-created-by-the-helm-chart-itself). +- [Increase the CPU / memory limits of the Cluster Prometheus](#increase-the-cpu--memory-limits-of-the-cluster-prometheus). + +## Ensure the cattle-monitoring-system namespace is placed into the System Project (or a similarly locked down Project that has access to other Projects in the cluster) + +Prometheus Operator's security model expects that the namespace it is deployed into (e.g., `cattle-monitoring-system`) has limited access for anyone except Cluster Admins to avoid privilege escalation via execing into Pods (such as the Jobs executing Helm operations). In addition, deploying Prometheus Federator and all Project Prometheus stacks into the System Project ensures that each Project Prometheus is able to reach out to scrape workloads across all Projects, even if Network Policies are defined via Project Network Isolation. It also provides limited access for Project Owners, Project Members, and other users so that they're unable to access data that they shouldn't have access to (i.e., being allowed to exec into pods, set up the ability to scrape namespaces outside of a given Project, etc.). + +## Configure rancher-monitoring to only watch for resources created by the Helm chart itself + +Since each Project Monitoring Stack will watch the other namespaces and collect additional custom workload metrics or dashboards already, it's recommended to configure the following settings on all selectors to ensure that the Cluster Prometheus Stack only monitors resources created by the Helm Chart itself: + +``` +matchLabels: + release: "rancher-monitoring" +``` + +The following selector fields are recommended to have this value: +- `.Values.alertmanager.alertmanagerSpec.alertmanagerConfigSelector` +- `.Values.prometheus.prometheusSpec.serviceMonitorSelector` +- `.Values.prometheus.prometheusSpec.podMonitorSelector` +- `.Values.prometheus.prometheusSpec.ruleSelector` +- `.Values.prometheus.prometheusSpec.probeSelector` + +Once this setting is turned on, you can always create ServiceMonitors or PodMonitors that are picked up by the Cluster Prometheus by adding the label `release: "rancher-monitoring"` to them, in which case they will be ignored by Project Monitoring Stacks automatically by default, even if the namespace in which those ServiceMonitors or PodMonitors reside in are not system namespaces. + +> Note: If you don't want to allow users to be able to create ServiceMonitors and PodMonitors that aggregate into the Cluster Prometheus in Project namespaces, you can additionally set the namespaceSelectors on the chart to only target system namespaces (which must contain `cattle-monitoring-system` and `cattle-dashboards`, where resources are deployed into by default by rancher-monitoring; you will also need to monitor the `default` namespace to get apiserver metrics or create a custom ServiceMonitor to scrape apiserver metrics from the Service residing in the default namespace) to limit your Cluster Prometheus from picking up other Prometheus Operator CRs. In that case, it would be recommended to turn `.Values.prometheus.prometheusSpec.ignoreNamespaceSelectors=true` to allow you to define ServiceMonitors that can monitor non-system namespaces from within a system namespace. + +## Increase the CPU / memory limits of the Cluster Prometheus + +Depending on a cluster's setup, it's generally recommended to give a large amount of dedicated memory to the Cluster Prometheus to avoid restarts due to out-of-memory errors (OOMKilled) usually caused by churn created in the cluster that causes a large number of high cardinality metrics to be generated and ingested by Prometheus within one block of time. This is one of the reasons why the default Rancher Monitoring stack expects around 4GB of RAM to be able to operate in a normal-sized cluster. However, when introducing Project Monitoring Stacks that are all sending `/federate` requests to the same Cluster Prometheus and are reliant on the Cluster Prometheus being "up" to federate that system data on their namespaces, it's even more important that the Cluster Prometheus has an ample amount of CPU / memory assigned to it to prevent an outage that can cause data gaps across all Project Prometheis in the cluster. + +> Note: There are no specific recommendations on how much memory the Cluster Prometheus should be configured with since it depends entirely on the user's setup (namely the likelihood of encountering a high churn rate and the scale of metrics that could be generated at that time); it generally varies per setup. + +# Install the Prometheus Federator Application + +1. Click **☰ > Cluster Management**. +1. Go to the cluster that you want to install Prometheus Federator and click **Explore**. +1. Click **Apps -> Charts**. +1. Click the **Prometheus Federator** chart. +1. Click **Install**. +1. On the **Metadata** page, click **Next**. +1. In the **Project Release Namespace Project ID** field, the `System Project` is used as the default but can be overridden with another project with similarly [limited access](#ensure-the-cattle-monitoring-system-namespace-is-placed-into-the-system-project-or-a-similarly-locked-down-project-that-has-access-to-other-projects-in-the-cluster). + +### Display CPU and Memory Metrics for a Workload + +Displaying CPU and memory metrics with Prometheus Federator is done the same way as with rancher-monitoring. For instructions, refer [here](../../../guides/monitoring-workloads/_index.md#display-cpu-and-memory-metrics-for-a-workload). + +### Setting up Metrics Beyond CPU and Memory + +Setting up metrics beyond CPU and memory with Prometheus Federator is done the same way as with rancher-monitoring. For instructions, refer [here](../../../guides/monitoring-workloads/_index.md#setting-up-metrics-beyond-cpu-and-memory). + + \ No newline at end of file diff --git a/content/rancher/v2.6/en/monitoring-alerting/prometheus-federator/guides/uninstall-prom-fed/_index.md b/content/rancher/v2.6/en/monitoring-alerting/prometheus-federator/guides/uninstall-prom-fed/_index.md new file mode 100644 index 00000000000..2db6cdda414 --- /dev/null +++ b/content/rancher/v2.6/en/monitoring-alerting/prometheus-federator/guides/uninstall-prom-fed/_index.md @@ -0,0 +1,14 @@ +--- +title: Uninstall Prometheus Federator +weight: 2 +--- + +1. Click **☰ > Cluster Management**. +1. Go to the cluster that you created and click **Explore**. +1. In the left navigation bar, click **Apps & Marketplace**. +1. Click **Installed Apps**. +1. Go to the `cattle-monitoring-system` namespace and check the boxes for `rancher-monitoring-crd` and `rancher-monitoring`. +1. Click **Delete**. +1. Confirm **Delete**. + +**Result:** `prometheus-federator` is uninstalled. diff --git a/content/rancher/v2.6/en/monitoring-alerting/prometheus-federator/rbac/_index.md b/content/rancher/v2.6/en/monitoring-alerting/prometheus-federator/rbac/_index.md new file mode 100644 index 00000000000..d656246d982 --- /dev/null +++ b/content/rancher/v2.6/en/monitoring-alerting/prometheus-federator/rbac/_index.md @@ -0,0 +1,29 @@ +--- +title: Role-Based Access Control +shortTitle: RBAC +weight: 2 +--- + +This section describes the expectations for Role-Based Access Control (RBAC) for Prometheus Federator. + +As described in the section on [namespaces](../_index.md#namespaces), Prometheus Federator expects that Project Owners, Project Members, and other users in the cluster with Project-level permissions (e.g. permissions in a certain set of namespaces identified by a single label selector) have minimal permissions in any namespaces except the Project Registration Namespace (which is imported into the project by default) and those that already comprise their projects. Therefore, in order to allow Project Owners to assign specific chart permissions to other users in their Project namespaces, the Helm Project Operator will automatically watch the following bindings: + +- ClusterRoleBindings +- RoleBindings in the Project Release Namespace + +On observing a change to one of those types of bindings, the Helm Project Operator will check whether the `roleRef` that the the binding points to matches a ClusterRole with the name provided under: + +- `helmProjectOperator.releaseRoleBindings.clusterRoleRefs.admin` +- `helmProjectOperator.releaseRoleBindings.clusterRoleRefs.edit` +- `helmProjectOperator.releaseRoleBindings.clusterRoleRefs.view` + +By default, these roleRefs will correspond to `admin`, `edit`, and `view` respectively, which are the [default Kubernetes user-facing roles](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#user-facing-roles). + +> **Note** For Rancher RBAC users, these [default Kubernetes user-facing roles](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#user-facing-roles) directly correlate to the `Project Owner`, `Project Member`, and `Read-Only` default Project Role Templates. + +If the `roleRef` matches, the Helm Project Operator will filter the `subjects` of the binding for all Users and Groups and use that to automatically construct a RoleBinding for each Role in the Project Release Namespace with the same name as the role and the following labels: + +- `helm.cattle.io/project-helm-chart-role: {{ .Release.Name }}` +- `helm.cattle.io/project-helm-chart-role-aggregate-from: ` + +By default, `rancher-project-monitoring`, the underlying chart deployed by Prometheus Federator, creates three default Roles per Project Release Namespace that provide `admin`, `edit`, and `view` users to permissions to view the Prometheus, Alertmanager, and Grafana UIs of the Project Monitoring Stack to provide least privilege. However, if a Cluster Admin would like to assign additional permissions to certain users, they can either directly assign RoleBindings in the Project Release Namespace to certain users or create Roles with the above two labels on them to allow Project Owners to control assigning those RBAC roles to users in their Project Registration namespaces. \ No newline at end of file diff --git a/content/rancher/v2.6/en/monitoring-alerting/rbac/_index.md b/content/rancher/v2.6/en/monitoring-alerting/rbac/_index.md index ba487525225..e025b733e64 100644 --- a/content/rancher/v2.6/en/monitoring-alerting/rbac/_index.md +++ b/content/rancher/v2.6/en/monitoring-alerting/rbac/_index.md @@ -124,8 +124,8 @@ The relationship between the default roles deployed by Rancher (i.e. cluster-own | --------- | --------- | --------- | --------- | | cluster-owner | cluster-admin | N/A | ClusterRoleBinding | | cluster-member | admin | monitoring-admin | ClusterRoleBinding | -| project-owner | edit | monitoring-admin | RoleBinding within Project namespace | -| project-member | view | monitoring-edit | RoleBinding within Project namespace | +| project-owner | admin | monitoring-admin | RoleBinding within Project namespace | +| project-member | edit | monitoring-edit | RoleBinding within Project namespace | In addition to these default Roles, the following additional Rancher project roles can be applied to members of your Cluster to provide additional access to Monitoring. These Rancher Roles will be tied to ClusterRoles deployed by the Monitoring chart: diff --git a/content/rancher/v2.6/en/neuvector-integration/_index.md b/content/rancher/v2.6/en/neuvector-integration/_index.md new file mode 100644 index 00000000000..1c58d6e200a --- /dev/null +++ b/content/rancher/v2.6/en/neuvector-integration/_index.md @@ -0,0 +1,120 @@ +--- +title: NeuVector Integration +weight: 22 +--- + +### NeuVector Integration in Rancher + +New in Rancher v2.6.5, [NeuVector 5.x](https://open-docs.neuvector.com/) is an open-source container-centric security platform that is now integrated into Rancher. NeuVector offers real-time compliance, visibility, and protection for critical applications and data during runtime. NeuVector provides a firewall, container process/file system monitoring, security auditing with CIS benchmarks, and vulnerability scanning. For more information on Rancher security, please see the [security documentation]({{}}/rancher/v2.6/en/security/). + +NeuVector can be enabled through a Helm chart that may be installed either through **Apps & Marketplace** or through the **Cluster Tools** button in the Rancher UI. Once the Helm chart is installed, users can easily [deploy and manage NeuVector clusters within Rancher](https://open-docs.neuvector.com/deploying/rancher#deploy-and-manage-neuvector-through-rancher-apps-marketplace). + +### Installing NeuVector with Rancher + +The Harvester Helm Chart is used to manage access to the NeuVector UI in Rancher where users can navigate directly to deploy and manage their NeuVector clusters. + +**To navigate to and install the NeuVector chart through Apps & Marketplace:** + +1. Click **☰ > Cluster Management**. +1. On the Clusters page, go to the cluster where you want to deploy NeuVector, and click **Explore**. +1. Go to **Apps & Marketplace > Charts**, and install **NeuVector** from the chart repo. +1. Different cluster types require different container runtimes. When configuring Helm chart values, go to the **Container Runtime** section, and select your runtime in accordance with the cluster type. Finally, click **Install** again. + +Some examples are as follows: + + - RKE1: `docker` + - K3s and RKE2: `k3scontainerd` + - AKS: `containerd` for v1.19 and up + - EKS: `docker` for v1.22 and below; `containerd` for v1.23 and up + - GKE: `containerd` (see the [Google docs](https://cloud.google.com/kubernetes-engine/docs/concepts/using-containerd) for more) + + >**Note:** Only one container runtime engine may be selected at a time during installation. + +**To navigate to and install the NeuVector chart through Cluster Tools:** + +1. Click **☰ > Cluster Management**. +1. On the Clusters page, go to the cluster where you want to deploy NeuVector, and click **Explore**. +1. Click on **Cluster Tools** at the bottom of the left navigation bar. +1. Repeat step 4 above to select your container runtime accordingly, then click **Install** again. + +### Accessing NeuVector from the Rancher UI + +1. Navigate to the cluster explorer of the cluster where NeuVector is installed. In the left navigation bar, click **NeuVector**. +1. Click the external link to go to the NeuVector UI. Once the link is selected, users must accept the `END USER LICENSE AGREEMENT` to access the NeuVector UI. + +### Uninstalling NeuVector from the Rancher UI + +**To uninstall from Apps & Marketplace:** + +1. Click **☰ > Cluster Management**. +1. Under **Apps & Marketplace**, click **Installed Apps**. +1. Under `cattle-neuvector-system`, select both the NeuVector app (and the associated CRD if desired), then click **Delete**. + +**To uninstall from Cluster Tools:** + +1. Click **☰ > Cluster Management**. +1. Click on **Cluster Tools** at the bottom-left of the screen, then click on the trash can icon under the NeuVector chart. Select `Delete the CRD associated with this app` if desired, then click **Delete**. + +### GitHub Repository + +The NeuVector project is available [here](https://github.com/neuvector/neuvector). + +### Documentation + +The NeuVector documentation is [here](https://open-docs.neuvector.com/). + +### Architecture + +The NeuVector security solution contains four types of security containers: Controllers, Enforcers, Managers, and Scanners. A special container called an All-in-One is also provided to combine the Controller, Enforcer, and Manager functions all in one container, primarily for Docker-native deployments. There is also an Updater which, when run, will update the CVE database. + +- **Controller:** Manages the NeuVector Enforcer container; provides REST APIs for the management console. +- **Enforcer:** Enforces security policies. +- **Manager:** Provides a web-UI and CLI console to manage the NeuVector platform. +- **All-in-One:** Includes the Controller, Enforcer, and Manager. +- **Scanner:** Performs the vulnerability and compliance scanning for images, containers, and nodes. +- **Updater:** Updates the CVE database for Neuvector (when run); redeploys scanner pods. + +
**NeuVector Security Containers:**
+![NeuVector Security Containers]({{}}/img/rancher/neuvector-security-containers.png) + +
**NeuVector Architecture:**
+![NeuVector Architecture]({{}}/img/rancher/neuvector-architecture.png) + +To learn more about NeuVector's architecture, please refer [here](https://open-docs.neuvector.com/basics/overview#architecture). + +### CPU and Memory Allocations + +Below are the minimum recommended computing resources for the NeuVector chart installation in a default deployment. Note that the resource limit is not set. + +| Container | CPU - Request | Memory - Request | +|------------|--------|---------| +| Controller | 3 (1GB 1vCPU needed per controller) | * +| Enforcer | On all nodes (500MB .5vCPU) | 1GB +| Manager | 1 (500MB .5vCPU) | * +| Scanner | 3 (100MB .5vCPU) | * + +\* Minimum 1GB of memory total required for Controller, Manager, and Scanner containers combined. + + +### Support Limitations + +* Only admins and cluster owners are currently supported. + +* Fleet multi-cluster deployment is not supported. + +* NeuVector is not supported on a Windows cluster. + +* NeuVector installation is not supported on hardened clusters. + +* NeuVector installation is not supported on SELinux clusters. + +* NeuVector installation is not supported on clusters in an air-gapped environment. + +### Other Limitations + +* Currently, NeuVector feature chart installation fails when a NeuVector partner chart already exists. To work around this issue, uninstall the NeuVector partner chart and reinstall the NeuVector feature chart. + +* Sometimes when the controllers are not ready, the NeuVector UI is not accessible from the Rancher UI. During this time, controllers will try to restart, and it takes a few minutes for the controllers to be active. + +* Container runtime is not auto-detected for different cluster types when installing the NeuVector chart. To work around this, you can specify the runtime manually. + diff --git a/content/rancher/v2.6/en/overview/_index.md b/content/rancher/v2.6/en/overview/_index.md index 69002e77d81..22d719374d7 100644 --- a/content/rancher/v2.6/en/overview/_index.md +++ b/content/rancher/v2.6/en/overview/_index.md @@ -38,7 +38,7 @@ The Rancher API server is built on top of an embedded Kubernetes API server and - **Provisioning Kubernetes clusters:** The Rancher API server can [provision Kubernetes]({{}}/rancher/v2.6/en/cluster-provisioning/) on existing nodes, or perform [Kubernetes upgrades.]({{}}/rancher/v2.6/en/cluster-admin/upgrading-kubernetes) - **Catalog management:** Rancher provides the ability to use a [catalog of Helm charts]({{}}/rancher/v2.6/en/helm-charts/) that make it easy to repeatedly deploy applications. -- **Managing projects:** A project is a group of multiple namespaces and access control policies within a cluster. A project is a Rancher concept, not a Kubernetes concept, which allows you manage multiple namespaces as a group and perform Kubernetes operations in them. The Rancher UI provides features for [project administration]({{}}/rancher/v2.6/en/project-admin/) and for [managing applications within projects.]({{}}/rancher/v2.6/en/k8s-in-rancher/) +- **Managing projects:** A project is a group of multiple namespaces and access control policies within a cluster. A project is a Rancher concept, not a Kubernetes concept, which allows you to manage multiple namespaces as a group and perform Kubernetes operations in them. The Rancher UI provides features for [project administration]({{}}/rancher/v2.6/en/project-admin/) and for [managing applications within projects.]({{}}/rancher/v2.6/en/k8s-in-rancher/) - **Pipelines:** Setting up a [pipeline]({{}}/rancher/v2.6/en/project-admin/pipelines/) can help developers deliver new software as quickly and efficiently as possible. Within Rancher, you can configure pipelines for each of your Rancher projects. - **Istio:** Our [integration with Istio]({{}}/rancher/v2.6/en/istio/) is designed so that a Rancher operator, such as an administrator or cluster owner, can deliver Istio to developers. Then developers can use Istio to enforce security policies, troubleshoot problems, or manage traffic for green/blue deployments, canary deployments, or A/B testing. diff --git a/content/rancher/v2.6/en/overview/architecture-recommendations/_index.md b/content/rancher/v2.6/en/overview/architecture-recommendations/_index.md index b679b3a3f72..112388c36c9 100644 --- a/content/rancher/v2.6/en/overview/architecture-recommendations/_index.md +++ b/content/rancher/v2.6/en/overview/architecture-recommendations/_index.md @@ -12,7 +12,7 @@ This section covers the following topics: - [Recommended Load Balancer Configuration for Kubernetes Installations](#recommended-load-balancer-configuration-for-kubernetes-installations) - [Environment for Kubernetes Installations](#environment-for-kubernetes-installations) - [Recommended Node Roles for Kubernetes Installations](#recommended-node-roles-for-kubernetes-installations) -- [Architecture for an Authorized Cluster Endpoint](#architecture-for-an-authorized-cluster-endpoint) +- [Architecture for an Authorized Cluster Endpoint (ACE)](#architecture-for-an-authorized-cluster-endpoint-ace) # Separation of Rancher and User Clusters @@ -32,7 +32,7 @@ We don't recommend installing Rancher in a single Docker container, because if t ### K3s Kubernetes Cluster Installations -One option for the underlying Kubernetes cluster is to use K3s Kubernetes. K3s Rancher's CNCF certified Kubernetes distribution. It is easy to install, with half the memory of Kubernetes, all in a binary less than 100 MB. One main advantage of this K3s architecture is that it allows an external datastore to hold the cluster data, allowing the K3s server nodes to be treated as ephemeral. +One option for the underlying Kubernetes cluster is to use K3s Kubernetes. K3s is Rancher's CNCF certified Kubernetes distribution. It is easy to install and uses half the memory of Kubernetes, all in a binary of less than 100 MB. Another advantage of K3s is that it allows an external datastore to hold the cluster data, allowing the K3s server nodes to be treated as ephemeral.
Architecture of a K3s Kubernetes Cluster Running the Rancher Management Server
![Architecture of a K3s Kubernetes Cluster Running the Rancher Management Server]({{}}/img/rancher/k3s-server-storage.svg) @@ -104,8 +104,10 @@ Because no additional workloads will be deployed on the Rancher server cluster, For more best practices for downstream clusters, refer to the [production checklist]({{}}/rancher/v2.6/en/cluster-provisioning/production) or our [best practices guide.]({{}}/rancher/v2.6/en/best-practices/) -# Architecture for an Authorized Cluster Endpoint +# Architecture for an Authorized Cluster Endpoint (ACE) -If you are using an [authorized cluster endpoint,]({{}}/rancher/v2.6/en/overview/architecture/#4-authorized-cluster-endpoint) we recommend creating an FQDN pointing to a load balancer which balances traffic across your nodes with the `controlplane` role. +If you are using an [authorized cluster endpoint (ACE),]({{}}/rancher/v2.6/en/overview/architecture/#4-authorized-cluster-endpoint) we recommend creating an FQDN pointing to a load balancer which balances traffic across your nodes with the `controlplane` role. If you are using private CA signed certificates on the load balancer, you have to supply the CA certificate, which will be included in the generated kubeconfig file to validate the certificate chain. See the documentation on [kubeconfig files]({{}}/rancher/v2.6/en/cluster-admin/cluster-access/kubectl/) and [API keys]({{}}/rancher/v2.6/en/user-settings/api-keys/#creating-an-api-key) for more information. + +As of Rancher v2.6.3, ACE support is available for registered RKE2 and K3s clusters. To view the manual steps to perform on the downstream cluster to enable the ACE, click [here]({{}}/rancher/v2.6/en/cluster-provisioning/registered-clusters/#authorized-cluster-endpoint-support-for-rke2-and-k3s-clusters). diff --git a/content/rancher/v2.6/en/overview/architecture/_index.md b/content/rancher/v2.6/en/overview/architecture/_index.md index 01d5be52f5f..a6c09157de9 100644 --- a/content/rancher/v2.6/en/overview/architecture/_index.md +++ b/content/rancher/v2.6/en/overview/architecture/_index.md @@ -20,7 +20,7 @@ This section covers the following topics: - [The authentication proxy](#1-the-authentication-proxy) - [Cluster controllers and cluster agents](#2-cluster-controllers-and-cluster-agents) - [Node agents](#3-node-agents) - - [Authorized cluster endpoint](#4-authorized-cluster-endpoint) + - [Authorized cluster endpoint (ACE)](#4-authorized-cluster-endpoint-ace) - [Important files](#important-files) - [Tools for provisioning Kubernetes clusters](#tools-for-provisioning-kubernetes-clusters) - [Rancher server components and source code](#rancher-server-components-and-source-code) @@ -103,11 +103,13 @@ If the cluster agent (also called `cattle-cluster-agent`) is not available, one The `cattle-node-agent` is deployed using a [DaemonSet](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/) resource to make sure it runs on every node in a Rancher-launched Kubernetes cluster. It is used to interact with the nodes when performing cluster operations. Examples of cluster operations include upgrading the Kubernetes version and creating or restoring etcd snapshots. -### 4. Authorized Cluster Endpoint +### 4. Authorized Cluster Endpoint (ACE) An authorized cluster endpoint allows users to connect to the Kubernetes API server of a downstream cluster without having to route their requests through the Rancher authentication proxy. -> The authorized cluster endpoint only works on Rancher-launched Kubernetes clusters. In other words, it only works in clusters where Rancher [used RKE]({{}}/rancher/v2.6/en/cluster-provisioning/rke-clusters) to provision the cluster. It is not available for registered clusters, or for clusters in a hosted Kubernetes provider, such as Amazon's EKS. +> The authorized cluster endpoint only works on Rancher-launched Kubernetes clusters. In other words, it only works in clusters where Rancher [used RKE]({{}}/rancher/v2.6/en/cluster-provisioning/rke-clusters) to provision the cluster. The ACE is not available for clusters in a hosted Kubernetes provider, such as Amazon's EKS. + +> The [ACE is available for registered RKE2 and K3s clusters]({{}}/rancher/v2.6/en/cluster-provisioning/registered-clusters/#authorized-cluster-endpoint-support-for-rke2-and-k3s-clusters) as of Rancher v2.6.3. There are two main reasons why a user might need the authorized cluster endpoint: diff --git a/content/rancher/v2.6/en/overview/concepts/_index.md b/content/rancher/v2.6/en/overview/concepts/_index.md index 2736a842f2d..5cd01b9460c 100644 --- a/content/rancher/v2.6/en/overview/concepts/_index.md +++ b/content/rancher/v2.6/en/overview/concepts/_index.md @@ -52,7 +52,7 @@ Three etcd nodes is generally sufficient for smaller clusters and five etcd node ### Controlplane Nodes -Controlplane nodes run the Kubernetes API server, scheduler, and controller manager. These nodes take care of routine tasks to ensure that your cluster maintains your configuration. Because all cluster data is stored on your etcd nodes, control plane nodes are stateless. You can run control plane on a single node, although two or more nodes are recommended for redundancy. Additionally, a single node can share the control plane and etcd roles. +Controlplane nodes run the Kubernetes API server, scheduler, and controller manager. These nodes take care of routine tasks to ensure that your cluster maintains your configuration. Because all cluster data is stored on your etcd nodes, control plane nodes are stateless. You can run control plane on a single node, although three or more nodes are recommended for redundancy. Additionally, a single node can share the control plane and etcd roles. ### Worker Nodes diff --git a/content/rancher/v2.6/en/pipelines/_index.md b/content/rancher/v2.6/en/pipelines/_index.md index ebf82d30aaf..5c9a2e868d7 100644 --- a/content/rancher/v2.6/en/pipelines/_index.md +++ b/content/rancher/v2.6/en/pipelines/_index.md @@ -3,7 +3,12 @@ title: Pipelines weight: 10 --- -> As of Rancher v2.5, Git-based deployment pipelines are now deprecated. We recommend handling pipelines with Rancher Continuous Delivery powered by [Fleet.](https://fleet.rancher.io/) To get to Fleet in Rancher, click ☰ > Continuous Delivery. Note that pipelines in Kubernetes 1.21+ are no longer supported. +> As of Rancher v2.5, Git-based deployment pipelines are now deprecated. We recommend handling pipelines with Rancher Continuous Delivery powered by [Fleet]({{}}/rancher/v2.6/en/deploy-across-clusters/fleet). To get to Fleet in Rancher, click ☰ > Continuous Delivery. +> +>**Notice:** +> +> - Pipelines in Kubernetes 1.21+ are no longer supported. +> - Fleet does not replace Rancher pipelines; the distinction is that Rancher pipelines are now powered by Fleet. Rancher's pipeline provides a simple CI/CD experience. Use it to automatically checkout code, run builds or scripts, publish Docker images or catalog applications, and deploy the updated software to users. diff --git a/content/rancher/v2.6/en/project-admin/project-members/_index.md b/content/rancher/v2.6/en/project-admin/project-members/_index.md index c84e655974f..32708445e60 100644 --- a/content/rancher/v2.6/en/project-admin/project-members/_index.md +++ b/content/rancher/v2.6/en/project-admin/project-members/_index.md @@ -38,7 +38,9 @@ Following project creation, you can add users as project members so that they ca >**Notes:** > - >- Users assigned the `Owner` or `Member` role for a project automatically inherit the `namespace creation` role. However, this role is a [Kubernetes ClusterRole](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#role-and-clusterrole), meaning its scope extends to all projects in the cluster. Therefore, users explicitly assigned the `Owner` or `Member` role for a project can create namespaces in other projects they're assigned to, even with only the `Read Only` role assigned. + >- Users assigned the `Owner` or `Member` role for a project automatically inherit the `namespace creation` role. However, this role is a [Kubernetes ClusterRole](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#role-and-clusterrole), meaning its scope extends to all projects in the cluster. Therefore, users explicitly assigned the `Owner` or `Member` role for a project can create or delete namespaces in other projects they're assigned to, even with only the `Read Only` role assigned. + > + >- By default, the Rancher role of `project-member` inherits from the `Kubernetes-edit` role, and the `project-owner` role inherits from the `Kubernetes-admin` role. As such, both `project-member` and `project-owner` roles will allow for namespace management, including the ability to create and delete namespaces. > >- For `Custom` roles, you can modify the list of individual roles available for assignment. > diff --git a/content/rancher/v2.6/en/project-admin/resource-quotas/_index.md b/content/rancher/v2.6/en/project-admin/resource-quotas/_index.md index 7b4239b9bdd..75e1aea6b90 100644 --- a/content/rancher/v2.6/en/project-admin/resource-quotas/_index.md +++ b/content/rancher/v2.6/en/project-admin/resource-quotas/_index.md @@ -38,4 +38,4 @@ Edit resource quotas when: 1. Click **Create**. -**Result:** The resource quota is applied to your project and namespaces. When you add more namespaces in the future, Rancher validates that the project can accommodate the namespace. If the project can't allocate the resources, Rancher won't let you save your changes. +**Result:** The resource quota is applied to your project and namespaces. When you add more namespaces in the future, Rancher validates that the project can accommodate the namespace. If the project can't allocate the resources, you may still create namespaces, but they will be given a resource quota of 0. Subsequently, Rancher will not allow you to create any resources restricted by this quota. diff --git a/content/rancher/v2.6/en/project-admin/resource-quotas/quotas-for-projects/_index.md b/content/rancher/v2.6/en/project-admin/resource-quotas/quotas-for-projects/_index.md index 63a18ba0f49..7e2aaf3e869 100644 --- a/content/rancher/v2.6/en/project-admin/resource-quotas/quotas-for-projects/_index.md +++ b/content/rancher/v2.6/en/project-admin/resource-quotas/quotas-for-projects/_index.md @@ -19,18 +19,38 @@ The resource quota includes two limits, which you set while creating or editing - **Project Limits:** - This set of values configures an overall resource limit for the project. If you try to add a new namespace to the project, Rancher uses the limits you've set to validate that the project has enough resources to accommodate the namespace. In other words, if you try to move a namespace into a project near its resource quota, Rancher blocks you from moving the namespace. + This set of values configures a total limit for each specified resource shared among all namespaces in the project. - **Namespace Default Limits:** - This value is the default resource limit available for each namespace. When the resource quota is created at the project level, this limit is automatically propagated to each namespace in the project. Each namespace is bound to this default limit unless you override it. + This set of values configures the default quota limit available for each namespace for each specified resource. + When a namespace is created in the project without overrides, this limit is automatically bound to the namespace and enforced. + In the following diagram, a Rancher administrator wants to apply a resource quota that sets the same CPU and memory limit for every namespace in their project (`Namespace 1-4`). However, in Rancher, the administrator can set a resource quota for the project (`Project Resource Quota`) rather than individual namespaces. This quota includes resource limits for both the entire project (`Project Limit`) and individual namespaces (`Namespace Default Limit`). Rancher then propagates the `Namespace Default Limit` quotas to each namespace (`Namespace Resource Quota`) when created. Rancher: Resource Quotas Propagating to Each Namespace ![Rancher Resource Quota Implementation]({{}}/img/rancher/rancher-resource-quota.png) -Let's highlight some more nuanced functionality. If a quota is deleted at the project level, it will also be removed from all namespaces contained within that project, despite any overrides that may exist. Further, updating an existing namespace default limit for a quota at the project level will not result in that value being propagated to existing namespaces in the project; the updated value will only be applied to newly created namespaces in that project. To update a namespace default limit for existing namespaces you can delete and subsequently recreate the quota at the project level with the new default value. This will result in the new default value being applied to all existing namespaces in the project. +Let's highlight some more nuanced functionality for namespaces created **_within_** the Rancher UI. If a quota is deleted at the project level, it will also be removed from all namespaces contained within that project, despite any overrides that may exist. Further, updating an existing namespace default limit for a quota at the project level will not result in that value being propagated to existing namespaces in the project; the updated value will only be applied to newly created namespaces in that project. To update a namespace default limit for existing namespaces you can delete and subsequently recreate the quota at the project level with the new default value. This will result in the new default value being applied to all existing namespaces in the project. + +Before creating a namespace in a project, Rancher compares the amounts of the project's available resources and requested resources, regardless of whether they come from the default or overridden limits. +If the requested resources exceed the remaining capacity in the project for those resources, Rancher will assign the namespace the remaining capacity for that resource. + +However, this is not the case with namespaces created **_outside_** of Rancher's UI. For namespaces created via `kubectl`, Rancher +will assign a resource quota that has a **zero** amount for any resource that requested more capacity than what remains in the project. + +To create a namespace in an existing project via `kubectl`, use the `field.cattle.io/projectId` annotation. To override the default +requested quota limit, use the `field.cattle.io/resourceQuota` annotation. +``` +apiVersion: v1 +kind: Namespace +metadata: + annotations: + field.cattle.io/projectId: [your-cluster-ID]:[your-project-ID] + field.cattle.io/resourceQuota: '{"limit":{"limitsCpu":"100m", "limitsMemory":"100Mi", "configMaps": "50"}}' + name: my-ns +``` The following table explains the key differences between the two quota types. diff --git a/content/rancher/v2.6/en/quick-start-guide/_index.md b/content/rancher/v2.6/en/quick-start-guide/_index.md index 65fa4db14f4..b6b0ba33866 100644 --- a/content/rancher/v2.6/en/quick-start-guide/_index.md +++ b/content/rancher/v2.6/en/quick-start-guide/_index.md @@ -1,8 +1,10 @@ --- title: Rancher Deployment Quick Start Guides metaDescription: Use this section to jump start your Rancher deployment and testing. It contains instructions for a simple Rancher setup and some common use cases. -short title: Use this section to jump start your Rancher deployment and testing. It contains instructions for a simple Rancher setup and some common use cases. +short title: Use this section to jump start your Rancher deployment and testing. It contains instructions for a simple Rancher setup and some common use cases. weight: 2 +aliases: + - /rancher/v2.x/en/quick-start-guide/ --- >**Note:** The intent of these guides is to quickly launch a sandbox that you can use to evaluate Rancher. These guides are not intended for production environments. For comprehensive setup instructions, see [Installation]({{}}/rancher/v2.6/en/installation/). @@ -13,5 +15,3 @@ We have Quick Start Guides for: - [Deploying Rancher Server]({{}}/rancher/v2.6/en/quick-start-guide/deployment/): Get started running Rancher using the method most convenient for you. - [Deploying Workloads]({{}}/rancher/v2.6/en/quick-start-guide/workload/): Deploy a simple [workload](https://kubernetes.io/docs/concepts/workloads/) and expose it, letting you access it from outside the cluster. - -- [Using the CLI]({{}}/rancher/v2.6/en/quick-start-guide/cli/): Use `kubectl` or Rancher command line interface (CLI) to interact with your Rancher instance. diff --git a/content/rancher/v2.6/en/quick-start-guide/cli/_index.md b/content/rancher/v2.6/en/quick-start-guide/cli/_index.md deleted file mode 100644 index c0787bc4c08..00000000000 --- a/content/rancher/v2.6/en/quick-start-guide/cli/_index.md +++ /dev/null @@ -1,39 +0,0 @@ ---- -title: CLI with Rancher -weight: 100 ---- - -Interact with Rancher using command line interface (CLI) tools from your workstation. - -## Rancher CLI - -Follow the steps in [rancher cli](../../cli). - -Ensure you can run `rancher kubectl get pods` successfully. - - -## kubectl -Install the `kubectl` utility. See [install kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/). - - -Configure kubectl by visiting your cluster in the Rancher Web UI then clicking on `Kubeconfig`, copying contents and putting into your `~/.kube/config` file. - -Run `kubectl cluster-info` or `kubectl get pods` successfully. - -## Authentication with kubectl and kubeconfig Tokens with TTL - -_Requirements_ - -If admins have [enforced TTL on kubeconfig tokens]({{}}/rancher/v2.6/en/api/api-tokens/#setting-ttl-on-kubeconfig-tokens), the kubeconfig file requires the [Rancher cli](../cli) to be present in your PATH when you run `kubectl`. Otherwise, you’ll see error like: -`Unable to connect to the server: getting credentials: exec: exec: "rancher": executable file not found in $PATH`. - -This feature enables kubectl to authenticate with the Rancher server and get a new kubeconfig token when required. The following auth providers are currently supported: - -1. Local -2. Active Directory -3. FreeIpa, OpenLdap -4. SAML providers - Ping, Okta, ADFS, Keycloak, Shibboleth - -When you first run kubectl, for example, `kubectl get pods`, it will ask you to pick an auth provider and log in with the Rancher server. -The kubeconfig token is cached in the path where you run kubectl under `./.cache/token`. This token is valid till [it expires](../../api/api-tokens/#setting-ttl-on-kubeconfig-tokens-period), or [gets deleted from the Rancher server](../../api/api-tokens/#deleting-tokens) -Upon expiration, the next `kubectl get pods` will ask you to log in with the Rancher server again. diff --git a/content/rancher/v2.6/en/quick-start-guide/deployment/_index.md b/content/rancher/v2.6/en/quick-start-guide/deployment/_index.md index f7d4da476aa..fa4236b3a6f 100644 --- a/content/rancher/v2.6/en/quick-start-guide/deployment/_index.md +++ b/content/rancher/v2.6/en/quick-start-guide/deployment/_index.md @@ -5,11 +5,15 @@ weight: 100 Use one of the following guides to deploy and provision Rancher and a Kubernetes cluster in the provider of your choice. -- [DigitalOcean](./digital-ocean-qs) (uses Terraform) - [AWS](./amazon-aws-qs) (uses Terraform) +- [AWS Marketplace](./amazon-aws-marketplace-qs) (uses Amazon EKS) - [Azure](./microsoft-azure-qs) (uses Terraform) +- [DigitalOcean](./digital-ocean-qs) (uses Terraform) - [GCP](./google-gcp-qs) (uses Terraform) +- [Hetzner Cloud](./hetzner-cloud-qs) (uses Terraform) - [Vagrant](./quickstart-vagrant) +- [Equinix Metal](./equinix-metal-qs) + If you prefer, the following guide will take you through the same process in individual steps. Use this if you want to run Rancher in a different provider, on prem, or if you would just like to see how easy it is. diff --git a/content/rancher/v2.6/en/quick-start-guide/deployment/amazon-aws-marketplace-qs/_index.md b/content/rancher/v2.6/en/quick-start-guide/deployment/amazon-aws-marketplace-qs/_index.md new file mode 100644 index 00000000000..bac47239b49 --- /dev/null +++ b/content/rancher/v2.6/en/quick-start-guide/deployment/amazon-aws-marketplace-qs/_index.md @@ -0,0 +1,7 @@ +--- +title: Rancher AWS Marketplace Quick Start +description: Use Amazon EKS to deploy Rancher server. +weight: 110 +--- + +There is now an additional way for you to deploy the Rancher server in AWS by using Amazon EKS. To learn more, see our [Amazon Marketplace listing](https://aws.amazon.com/marketplace/pp/prodview-2yzbnvagmi4as). \ No newline at end of file diff --git a/content/rancher/v2.6/en/quick-start-guide/deployment/amazon-aws-qs/_index.md b/content/rancher/v2.6/en/quick-start-guide/deployment/amazon-aws-qs/_index.md index 1a0bc66ec49..64ba81e19bf 100644 --- a/content/rancher/v2.6/en/quick-start-guide/deployment/amazon-aws-qs/_index.md +++ b/content/rancher/v2.6/en/quick-start-guide/deployment/amazon-aws-qs/_index.md @@ -1,9 +1,11 @@ --- title: Rancher AWS Quick Start Guide -description: Read this step by step Rancher AWS guide to quickly deploy a Rancher Server with a single node cluster attached. +description: Read this step by step Rancher AWS guide to quickly deploy a Rancher server with a single-node downstream Kubernetes cluster attached. weight: 100 --- -The following steps will quickly deploy a Rancher Server on AWS with a single node cluster attached. +The following steps will quickly deploy a Rancher server on AWS in a single-node K3s Kubernetes cluster, with a single-node downstream Kubernetes cluster attached. + +>**Note:** The intent of these guides is to quickly launch a sandbox that you can use to evaluate Rancher. These guides are not intended for production environments. For comprehensive setup instructions, see [Installation]({{}}/rancher/v2.6/en/installation/). ## Prerequisites @@ -19,25 +21,26 @@ The following steps will quickly deploy a Rancher Server on AWS with a single no 1. Clone [Rancher Quickstart](https://github.com/rancher/quickstart) to a folder using `git clone https://github.com/rancher/quickstart`. -1. Go into the AWS folder containing the terraform files by executing `cd quickstart/aws`. +2. Go into the AWS folder containing the terraform files by executing `cd quickstart/aws`. -1. Rename the `terraform.tfvars.example` file to `terraform.tfvars`. +3. Rename the `terraform.tfvars.example` file to `terraform.tfvars`. -1. Edit `terraform.tfvars` and customize the following variables: - - `aws_access_key` - Amazon AWS Access Key +4. Edit `terraform.tfvars` and customize the following variables: + - `aws_access_key` - Amazon AWS Access Key - `aws_secret_key` - Amazon AWS Secret Key - `rancher_server_admin_password` - Admin password for created Rancher server -1. **Optional:** Modify optional variables within `terraform.tfvars`. +5. **Optional:** Modify optional variables within `terraform.tfvars`. See the [Quickstart Readme](https://github.com/rancher/quickstart) and the [AWS Quickstart Readme](https://github.com/rancher/quickstart/tree/master/aws) for more information. Suggestions include: - - `aws_region` - Amazon AWS region, choose the closest instead of the default + - `aws_region` - Amazon AWS region, choose the closest instead of the default (`us-east-1`) - `prefix` - Prefix for all created resources - `instance_type` - EC2 instance size used, minimum is `t3a.medium` but `t3a.large` or `t3a.xlarge` could be used if within budget + - `add_windows_node` - If true, an additional Windows worker node is added to the workload cluster -1. Run `terraform init`. +6. Run `terraform init`. -1. To initiate the creation of the environment, run `terraform apply --auto-approve`. Then wait for output similar to the following: +7. To initiate the creation of the environment, run `terraform apply --auto-approve`. Then wait for output similar to the following: ``` Apply complete! Resources: 16 added, 0 changed, 0 destroyed. @@ -45,17 +48,18 @@ Suggestions include: Outputs: rancher_node_ip = xx.xx.xx.xx - rancher_server_url = https://rancher.xx.xx.xx.xx.xip.io + rancher_server_url = https://rancher.xx.xx.xx.xx.sslip.io workload_node_ip = yy.yy.yy.yy ``` -1. Paste the `rancher_server_url` from the output above into the browser. Log in when prompted (default username is `admin`, use the password set in `rancher_server_admin_password`). +8. Paste the `rancher_server_url` from the output above into the browser. Log in when prompted (default username is `admin`, use the password set in `rancher_server_admin_password`). +9. ssh to the Rancher Server using the `id_rsa` key generated in `quickstart/aws`. -#### Result +##### Result -Two Kubernetes clusters are deployed into your AWS account, one running Rancher Server and the other ready for experimentation deployments. Please note that while this setup is a great way to explore Rancher functionality, a production setup should follow our high availability setup guidelines. +Two Kubernetes clusters are deployed into your AWS account, one running Rancher Server and the other ready for experimentation deployments. Please note that while this setup is a great way to explore Rancher functionality, a production setup should follow our high availability setup guidelines. SSH keys for the VMs are auto-generated and stored in the module directory. -### What's Next? +## What's Next? Use Rancher to create a deployment. For more information, see [Creating Deployments]({{}}/rancher/v2.6/en/quick-start-guide/workload). diff --git a/content/rancher/v2.6/en/quick-start-guide/deployment/digital-ocean-qs/_index.md b/content/rancher/v2.6/en/quick-start-guide/deployment/digital-ocean-qs/_index.md index 02384c1b601..8afb42b9d33 100644 --- a/content/rancher/v2.6/en/quick-start-guide/deployment/digital-ocean-qs/_index.md +++ b/content/rancher/v2.6/en/quick-start-guide/deployment/digital-ocean-qs/_index.md @@ -1,9 +1,11 @@ --- title: Rancher DigitalOcean Quick Start Guide -description: Read this step by step Rancher DigitalOcean guide to quickly deploy a Rancher Server with a single node cluster attached. -weight: 100 +description: Read this step by step Rancher DigitalOcean guide to quickly deploy a Rancher server with a single-node downstream Kubernetes cluster attached. +weight: 120 --- -The following steps will quickly deploy a Rancher Server on DigitalOcean with a single node cluster attached. +The following steps will quickly deploy a Rancher server on DigitalOcean in a single-node K3s Kubernetes cluster, with a single-node downstream Kubernetes cluster attached. + +>**Note:** The intent of these guides is to quickly launch a sandbox that you can use to evaluate Rancher. These guides are not intended for production environments. For comprehensive setup instructions, see [Installation]({{}}/rancher/v2.6/en/installation/). ## Prerequisites @@ -19,25 +21,24 @@ The following steps will quickly deploy a Rancher Server on DigitalOcean with a 1. Clone [Rancher Quickstart](https://github.com/rancher/quickstart) to a folder using `git clone https://github.com/rancher/quickstart`. -1. Go into the DigitalOcean folder containing the terraform files by executing `cd quickstart/do`. +2. Go into the DigitalOcean folder containing the terraform files by executing `cd quickstart/do`. -1. Rename the `terraform.tfvars.example` file to `terraform.tfvars`. +3. Rename the `terraform.tfvars.example` file to `terraform.tfvars`. -1. Edit `terraform.tfvars` and customize the following variables: +4. Edit `terraform.tfvars` and customize the following variables: - `do_token` - DigitalOcean access key - `rancher_server_admin_password` - Admin password for created Rancher server -1. **Optional:** Modify optional variables within `terraform.tfvars`. +5. **Optional:** Modify optional variables within `terraform.tfvars`. See the [Quickstart Readme](https://github.com/rancher/quickstart) and the [DO Quickstart Readme](https://github.com/rancher/quickstart/tree/master/do) for more information. Suggestions include: - - `do_region` - DigitalOcean region, choose the closest instead of the default + - `do_region` - DigitalOcean region, choose the closest instead of the default (`nyc1`) - `prefix` - Prefix for all created resources - `droplet_size` - Droplet size used, minimum is `s-2vcpu-4gb` but `s-4vcpu-8gb` could be used if within budget - - `ssh_key_file_name` - Use a specific SSH key instead of `~/.ssh/id_rsa` (public key is assumed to be `${ssh_key_file_name}.pub`) -1. Run `terraform init`. +6. Run `terraform init`. -1. To initiate the creation of the environment, run `terraform apply --auto-approve`. Then wait for output similar to the following: +7. To initiate the creation of the environment, run `terraform apply --auto-approve`. Then wait for output similar to the following: ``` Apply complete! Resources: 15 added, 0 changed, 0 destroyed. @@ -45,15 +46,16 @@ Suggestions include: Outputs: rancher_node_ip = xx.xx.xx.xx - rancher_server_url = https://rancher.xx.xx.xx.xx.xip.io + rancher_server_url = https://rancher.xx.xx.xx.xx.sslip.io workload_node_ip = yy.yy.yy.yy ``` -1. Paste the `rancher_server_url` from the output above into the browser. Log in when prompted (default username is `admin`, use the password set in `rancher_server_admin_password`). +8. Paste the `rancher_server_url` from the output above into the browser. Log in when prompted (default username is `admin`, use the password set in `rancher_server_admin_password`). +9. ssh to the Rancher Server using the `id_rsa` key generated in `quickstart/do`. #### Result -Two Kubernetes clusters are deployed into your DigitalOcean account, one running Rancher Server and the other ready for experimentation deployments. +Two Kubernetes clusters are deployed into your DigitalOcean account, one running Rancher Server and the other ready for experimentation deployments. Please note that while this setup is a great way to explore Rancher functionality, a production setup should follow our high availability setup guidelines. SSH keys for the VMs are auto-generated and stored in the module directory. ### What's Next? diff --git a/content/rancher/v2.6/en/quick-start-guide/deployment/equinix-metal-qs/_index.md b/content/rancher/v2.6/en/quick-start-guide/deployment/equinix-metal-qs/_index.md index 0daf725fdbc..ad208bd3bf8 100644 --- a/content/rancher/v2.6/en/quick-start-guide/deployment/equinix-metal-qs/_index.md +++ b/content/rancher/v2.6/en/quick-start-guide/deployment/equinix-metal-qs/_index.md @@ -1,6 +1,6 @@ --- title: Rancher Equinix Metal Quick Start -weight: 300 +weight: 250 --- ## This tutorial walks you through the following: @@ -10,6 +10,8 @@ weight: 300 - Creation of your first cluster - Deployment of an application, Nginx +>**Note:** The intent of these guides is to quickly launch a sandbox that you can use to evaluate Rancher. The Docker install is not recommended for production environments. For comprehensive setup instructions, see [Installation]({{}}/rancher/v2.6/en/installation/). + ## Quick Start Outline This Quick Start Guide is divided into different tasks for easier consumption. @@ -41,7 +43,7 @@ This Quick Start Guide is divided into different tasks for easier consumption. - [Equinix Metal Pricing](https://metal.equinix.com/developers/docs/servers/server-specs/) **Note:** - > When provisioning a new Equinix Metal Server via the CLI or API you will need to be able to provide the following information: project-id, plan, metro, and the operating-system + > When provisioning a new Equinix Metal Server via the CLI or API you will need to be able to provide the following information: project-id, plan, metro, and the operating-system > When using a cloud-hosted virtual machine you need to allow inbound TCP communication to ports 80 and 443. Please see your cloud-host's documentation for information regarding port configuration. > For a full list of port requirements, refer to [Docker Installation]({{}}/rancher/v2.6/en/cluster-provisioning/node-requirements/). > Provision the host according to our [Requirements]({{}}/rancher/v2.6/en/installation/requirements/). @@ -102,13 +104,13 @@ In this task, you can use the versatile **Custom** option. This option lets you 11. When you finish running the command on your Linux host, click **Done**. -**Result:** +**Result:** Your cluster is created and assigned a state of **Provisioning**. Rancher is standing up your cluster. You can access your cluster after its state is updated to **Active**. -**Active** clusters are assigned two Projects: +**Active** clusters are assigned two Projects: - `Default`, containing the `default` namespace - `System`, containing the `cattle-system`, `ingress-nginx`, `kube-public`, and `kube-system` namespaces diff --git a/content/rancher/v2.6/en/quick-start-guide/deployment/google-gcp-qs/_index.md b/content/rancher/v2.6/en/quick-start-guide/deployment/google-gcp-qs/_index.md index fe13ba86aca..b589f810c46 100644 --- a/content/rancher/v2.6/en/quick-start-guide/deployment/google-gcp-qs/_index.md +++ b/content/rancher/v2.6/en/quick-start-guide/deployment/google-gcp-qs/_index.md @@ -1,67 +1,70 @@ ---- -title: Rancher GCP Quick Start Guide -description: Read this step by step Rancher GCP guide to quickly deploy a Rancher Server with a single node cluster attached. -weight: 100 ---- -The following steps will quickly deploy a Rancher server on GCP in a single-node RKE Kubernetes cluster, with a single-node downstream Kubernetes cluster attached. - -## Prerequisites - ->**Note** ->Deploying to Google GCP will incur charges. - -- [Google GCP Account](https://console.cloud.google.com/): A Google GCP Account is required to create resources for deploying Rancher and Kubernetes. -- [Google GCP Project](https://cloud.google.com/appengine/docs/standard/nodejs/building-app/creating-project): Use this link to follow a tutorial to create a GCP Project if you don't have one yet. -- [Google GCP Service Account](https://cloud.google.com/iam/docs/creating-managing-service-account-keys): Use this link and follow instructions to create a GCP service account and token file. -- [Terraform](https://www.terraform.io/downloads.html): Used to provision the server and cluster in Google GCP. - - -## Getting Started - -1. Clone [Rancher Quickstart](https://github.com/rancher/quickstart) to a folder using `git clone https://github.com/rancher/quickstart`. - -1. Go into the GCP folder containing the terraform files by executing `cd quickstart/gcp`. - -1. Rename the `terraform.tfvars.example` file to `terraform.tfvars`. - -1. Edit `terraform.tfvars` and customize the following variables: - - `gcp_account_json` - GCP service account file path and file name - - `rancher_server_admin_password` - Admin password for created Rancher server - -1. **Optional:** Modify optional variables within `terraform.tfvars`. -See the [Quickstart Readme](https://github.com/rancher/quickstart) and the [GCP Quickstart Readme](https://github.com/rancher/quickstart/tree/master/gcp) for more information. -Suggestions include: - - `gcp_region` - Google GCP region, choose the closest instead of the default - - `prefix` - Prefix for all created resources - - `machine_type` - Compute instance size used, minimum is `n1-standard-1` but `n1-standard-2` or `n1-standard-4` could be used if within budget - - `ssh_key_file_name` - Use a specific SSH key instead of `~/.ssh/id_rsa` (public key is assumed to be `${ssh_key_file_name}.pub`) - -1. Run `terraform init`. - -1. To initiate the creation of the environment, run `terraform apply --auto-approve`. Then wait for output similar to the following: - - ``` - Apply complete! Resources: 16 added, 0 changed, 0 destroyed. - - Outputs: - - rancher_node_ip = xx.xx.xx.xx - rancher_server_url = https://rancher.xx.xx.xx.xx.xip.io - workload_node_ip = yy.yy.yy.yy - ``` - -1. Paste the `rancher_server_url` from the output above into the browser. Log in when prompted (default username is `admin`, use the password set in `rancher_server_admin_password`). - -#### Result - -Two Kubernetes clusters are deployed into your GCP account, one running Rancher Server and the other ready for experimentation deployments. - -### What's Next? - -Use Rancher to create a deployment. For more information, see [Creating Deployments]({{< baseurl >}}/rancher/v2.6/en/quick-start-guide/workload). - -## Destroying the Environment - -1. From the `quickstart/gcp` folder, execute `terraform destroy --auto-approve`. - -2. Wait for confirmation that all resources have been destroyed. +--- +title: Rancher GCP Quick Start Guide +description: Read this step by step Rancher GCP guide to quickly deploy a Rancher server with a single-node downstream Kubernetes cluster attached. +weight: 130 +--- +The following steps will quickly deploy a Rancher server on GCP in a single-node K3s Kubernetes cluster, with a single-node downstream Kubernetes cluster attached. + +>**Note:** The intent of these guides is to quickly launch a sandbox that you can use to evaluate Rancher. These guides are not intended for production environments. For comprehensive setup instructions, see [Installation]({{}}/rancher/v2.6/en/installation/). + +## Prerequisites + +>**Note** +>Deploying to Google GCP will incur charges. + +- [Google GCP Account](https://console.cloud.google.com/): A Google GCP Account is required to create resources for deploying Rancher and Kubernetes. +- [Google GCP Project](https://cloud.google.com/appengine/docs/standard/nodejs/building-app/creating-project): Use this link to follow a tutorial to create a GCP Project if you don't have one yet. +- [Google GCP Service Account](https://cloud.google.com/iam/docs/creating-managing-service-account-keys): Use this link and follow instructions to create a GCP service account and token file. +- [Terraform](https://www.terraform.io/downloads.html): Used to provision the server and cluster in Google GCP. + + +## Getting Started + +1. Clone [Rancher Quickstart](https://github.com/rancher/quickstart) to a folder using `git clone https://github.com/rancher/quickstart`. + +2. Go into the GCP folder containing the terraform files by executing `cd quickstart/gcp`. + +3. Rename the `terraform.tfvars.example` file to `terraform.tfvars`. + +4. Edit `terraform.tfvars` and customize the following variables: + - `gcp_account_json` - GCP service account file path and file name + - `rancher_server_admin_password` - Admin password for created Rancher server + +5. **Optional:** Modify optional variables within `terraform.tfvars`. +See the [Quickstart Readme](https://github.com/rancher/quickstart) and the [GCP Quickstart Readme](https://github.com/rancher/quickstart/tree/master/gcp) for more information. +Suggestions include: + - `gcp_region` - Google GCP region, choose the closest instead of the default (`us-east4`) + - `gcp_zone` - Google GCP zone, choose the closest instead of the default (`us-east4-a`) + - `prefix` - Prefix for all created resources + - `machine_type` - Compute instance size used, minimum is `n1-standard-1` but `n1-standard-2` or `n1-standard-4` could be used if within budget + +6. Run `terraform init`. + +7. To initiate the creation of the environment, run `terraform apply --auto-approve`. Then wait for output similar to the following: + + ``` + Apply complete! Resources: 16 added, 0 changed, 0 destroyed. + + Outputs: + + rancher_node_ip = xx.xx.xx.xx + rancher_server_url = https://rancher.xx.xx.xx.xx.sslip.io + workload_node_ip = yy.yy.yy.yy + ``` + +8. Paste the `rancher_server_url` from the output above into the browser. Log in when prompted (default username is `admin`, use the password set in `rancher_server_admin_password`). +9. ssh to the Rancher Server using the `id_rsa` key generated in `quickstart/gcp`. + +#### Result + +Two Kubernetes clusters are deployed into your GCP account, one running Rancher Server and the other ready for experimentation deployments. Please note that while this setup is a great way to explore Rancher functionality, a production setup should follow our high availability setup guidelines. SSH keys for the VMs are auto-generated and stored in the module directory. + +### What's Next? + +Use Rancher to create a deployment. For more information, see [Creating Deployments]({{< baseurl >}}/rancher/v2.6/en/quick-start-guide/workload). + +## Destroying the Environment + +1. From the `quickstart/gcp` folder, execute `terraform destroy --auto-approve`. + +2. Wait for confirmation that all resources have been destroyed. diff --git a/content/rancher/v2.6/en/quick-start-guide/deployment/hetzner-cloud-qs/_index.md b/content/rancher/v2.6/en/quick-start-guide/deployment/hetzner-cloud-qs/_index.md new file mode 100644 index 00000000000..ea7a2bbf1b8 --- /dev/null +++ b/content/rancher/v2.6/en/quick-start-guide/deployment/hetzner-cloud-qs/_index.md @@ -0,0 +1,68 @@ +--- +title: Rancher Hetzner Cloud Quick Start Guide +description: Read this step by step Rancher Hetzner Cloud guide to quickly deploy a Rancher server with a single-node downstream Kubernetes cluster attached. +weight: 140 +--- +The following steps will quickly deploy a Rancher server on Hetzner Cloud in a single-node K3s Kubernetes cluster, with a single-node downstream Kubernetes cluster attached. + +>**Note:** The intent of these guides is to quickly launch a sandbox that you can use to evaluate Rancher. These guides are not intended for production environments. For comprehensive setup instructions, see [Installation]({{}}/rancher/v2.6/en/installation/). + +## Prerequisites + +>**Note** +>Deploying to Hetzner Cloud will incur charges. + +- [Hetzner Cloud Account](https://www.hetzner.com): You will require an account on Hetzner as this is where the server and cluster will run. +- [Hetzner API Access Key](https://docs.hetzner.cloud/#getting-started): Use these instructions to create a Hetzner Cloud API Key if you don't have one. +- [Terraform](https://www.terraform.io/downloads.html): Used to provision the server and cluster to Hetzner. + + +## Getting Started + +1. Clone [Rancher Quickstart](https://github.com/rancher/quickstart) to a folder using `git clone https://github.com/rancher/quickstart`. + +2. Go into the Hetzner folder containing the terraform files by executing `cd quickstart/hcloud`. + +3. Rename the `terraform.tfvars.example` file to `terraform.tfvars`. + +4. Edit `terraform.tfvars` and customize the following variables: + - `hcloud_token` - Hetzner API access key + - `rancher_server_admin_password` - Admin password for created Rancher server + +5. **Optional:** Modify optional variables within `terraform.tfvars`. +See the [Quickstart Readme](https://github.com/rancher/quickstart) and the [Hetzner Quickstart Readme](https://github.com/rancher/quickstart/tree/master/hcloud) for more information. +Suggestions include: + - `prefix` - Prefix for all created resources + - `instance_type` - Instance type, minimum required is `cx21` + - `hcloud_location` - Hetzner Cloud location, choose the closest instead of the default (`fsn1`) + +6. Run `terraform init`. + +7. To initiate the creation of the environment, run `terraform apply --auto-approve`. Then wait for output similar to the following: + + ``` + Apply complete! Resources: 15 added, 0 changed, 0 destroyed. + + Outputs: + + rancher_node_ip = xx.xx.xx.xx + rancher_server_url = https://rancher.xx.xx.xx.xx.sslip.io + workload_node_ip = yy.yy.yy.yy + ``` + +8. Paste the `rancher_server_url` from the output above into the browser. Log in when prompted (default username is `admin`, use the password set in `rancher_server_admin_password`). +9. ssh to the Rancher Server using the `id_rsa` key generated in `quickstart/hcloud`. + +#### Result + +Two Kubernetes clusters are deployed into your Hetzner account, one running Rancher Server and the other ready for experimentation deployments. Please note that while this setup is a great way to explore Rancher functionality, a production setup should follow our high availability setup guidelines. SSH keys for the VMs are auto-generated and stored in the module directory. + +### What's Next? + +Use Rancher to create a deployment. For more information, see [Creating Deployments]({{}}/rancher/v2.6/en/quick-start-guide/workload). + +## Destroying the Environment + +1. From the `quickstart/hcloud` folder, execute `terraform destroy --auto-approve`. + +2. Wait for confirmation that all resources have been destroyed. diff --git a/content/rancher/v2.6/en/quick-start-guide/deployment/microsoft-azure-qs/_index.md b/content/rancher/v2.6/en/quick-start-guide/deployment/microsoft-azure-qs/_index.md index c3d758f9ccf..ec8d16a475e 100644 --- a/content/rancher/v2.6/en/quick-start-guide/deployment/microsoft-azure-qs/_index.md +++ b/content/rancher/v2.6/en/quick-start-guide/deployment/microsoft-azure-qs/_index.md @@ -1,71 +1,76 @@ ---- -title: Rancher Azure Quick Start Guide -description: Read this step by step Rancher Azure guide to quickly deploy a Rancher Server with a single node cluster attached. -weight: 100 ---- - -The following steps will quickly deploy a Rancher server on Azure in a single-node RKE Kubernetes cluster, with a single-node downstream Kubernetes cluster attached. - -## Prerequisites - ->**Note** ->Deploying to Microsoft Azure will incur charges. - -- [Microsoft Azure Account](https://azure.microsoft.com/en-us/free/): A Microsoft Azure Account is required to create resources for deploying Rancher and Kubernetes. -- [Microsoft Azure Subscription](https://docs.microsoft.com/en-us/azure/cost-management-billing/manage/create-subscription#create-a-subscription-in-the-azure-portal): Use this link to follow a tutorial to create a Microsoft Azure subscription if you don't have one yet. -- [Micsoroft Azure Tenant](https://docs.microsoft.com/en-us/azure/active-directory/develop/quickstart-create-new-tenant): Use this link and follow instructions to create a Microsoft Azure tenant. -- [Microsoft Azure Client ID/Secret](https://docs.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal): Use this link and follow instructions to create a Microsoft Azure client and secret. -- [Terraform](https://www.terraform.io/downloads.html): Used to provision the server and cluster in Microsoft Azure. - - -## Getting Started - -1. Clone [Rancher Quickstart](https://github.com/rancher/quickstart) to a folder using `git clone https://github.com/rancher/quickstart`. - -1. Go into the Azure folder containing the terraform files by executing `cd quickstart/azure`. - -1. Rename the `terraform.tfvars.example` file to `terraform.tfvars`. - -1. Edit `terraform.tfvars` and customize the following variables: - - `azure_subscription_id` - Microsoft Azure Subscription ID - - `azure_client_id` - Microsoft Azure Client ID - - `azure_client_secret` - Microsoft Azure Client Secret - - `azure_tenant_id` - Microsoft Azure Tenant ID - - `rancher_server_admin_password` - Admin password for created Rancher server - -2. **Optional:** Modify optional variables within `terraform.tfvars`. -See the [Quickstart Readme](https://github.com/rancher/quickstart) and the [Azure Quickstart Readme](https://github.com/rancher/quickstart/tree/master/azure) for more information. -Suggestions include: - - `azure_location` - Microsoft Azure region, choose the closest instead of the default - - `prefix` - Prefix for all created resources - - `instance_type` - Compute instance size used, minimum is `Standard_DS2_v2` but `Standard_DS2_v3` or `Standard_DS3_v2` could be used if within budget - -1. Run `terraform init`. - -1. To initiate the creation of the environment, run `terraform apply --auto-approve`. Then wait for output similar to the following: - - ``` - Apply complete! Resources: 16 added, 0 changed, 0 destroyed. - - Outputs: - - rancher_node_ip = xx.xx.xx.xx - rancher_server_url = https://rancher.xx.xx.xx.xx.xip.io - workload_node_ip = yy.yy.yy.yy - ``` - -1. Paste the `rancher_server_url` from the output above into the browser. Log in when prompted (default username is `admin`, use the password set in `rancher_server_admin_password`). -2. ssh to the Rancher Server using the `id_rsa` key generated in `quickstart/azure`. -#### Result - -Two Kubernetes clusters are deployed into your Azure account, one running Rancher Server and the other ready for experimentation deployments. - -### What's Next? - -Use Rancher to create a deployment. For more information, see [Creating Deployments]({{< baseurl >}}/rancher/v2.6/en/quick-start-guide/workload). - -## Destroying the Environment - -1. From the `quickstart/azure` folder, execute `terraform destroy --auto-approve`. - -2. Wait for confirmation that all resources have been destroyed. +--- +title: Rancher Azure Quick Start Guide +description: Read this step by step Rancher Azure guide to quickly deploy a Rancher server with a single-node downstream Kubernetes cluster attached. +weight: 115 +--- + +The following steps will quickly deploy a Rancher server on Azure in a single-node K3s Kubernetes cluster, with a single-node downstream Kubernetes cluster attached. + +>**Note:** The intent of these guides is to quickly launch a sandbox that you can use to evaluate Rancher. These guides are not intended for production environments. For comprehensive setup instructions, see [Installation]({{}}/rancher/v2.6/en/installation/). + +## Prerequisites + +>**Note** +>Deploying to Microsoft Azure will incur charges. + +- [Microsoft Azure Account](https://azure.microsoft.com/en-us/free/): A Microsoft Azure Account is required to create resources for deploying Rancher and Kubernetes. +- [Microsoft Azure Subscription](https://docs.microsoft.com/en-us/azure/cost-management-billing/manage/create-subscription#create-a-subscription-in-the-azure-portal): Use this link to follow a tutorial to create a Microsoft Azure subscription if you don't have one yet. +- [Micsoroft Azure Tenant](https://docs.microsoft.com/en-us/azure/active-directory/develop/quickstart-create-new-tenant): Use this link and follow instructions to create a Microsoft Azure tenant. +- [Microsoft Azure Client ID/Secret](https://docs.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal): Use this link and follow instructions to create a Microsoft Azure client and secret. +- [Terraform](https://www.terraform.io/downloads.html): Used to provision the server and cluster in Microsoft Azure. + + +## Getting Started + +1. Clone [Rancher Quickstart](https://github.com/rancher/quickstart) to a folder using `git clone https://github.com/rancher/quickstart`. + +2. Go into the Azure folder containing the terraform files by executing `cd quickstart/azure`. + +3. Rename the `terraform.tfvars.example` file to `terraform.tfvars`. + +4. Edit `terraform.tfvars` and customize the following variables: + - `azure_subscription_id` - Microsoft Azure Subscription ID + - `azure_client_id` - Microsoft Azure Client ID + - `azure_client_secret` - Microsoft Azure Client Secret + - `azure_tenant_id` - Microsoft Azure Tenant ID + - `rancher_server_admin_password` - Admin password for created Rancher server + +5. **Optional:** Modify optional variables within `terraform.tfvars`. +See the [Quickstart Readme](https://github.com/rancher/quickstart) and the [Azure Quickstart Readme](https://github.com/rancher/quickstart/tree/master/azure) for more information. +Suggestions include: + - `azure_location` - Microsoft Azure region, choose the closest instead of the default (`East US`) + - `prefix` - Prefix for all created resources + - `instance_type` - Compute instance size used, minimum is `Standard_DS2_v2` but `Standard_DS2_v3` or `Standard_DS3_v2` could be used if within budget + - `add_windows_node` - If true, an additional Windows worker node is added to the workload cluster + - `windows_admin_password` - The admin password of the windows worker node + +6. Run `terraform init`. + +7. To initiate the creation of the environment, run `terraform apply --auto-approve`. Then wait for output similar to the following: + + ``` + Apply complete! Resources: 16 added, 0 changed, 0 destroyed. + + Outputs: + + rancher_node_ip = xx.xx.xx.xx + rancher_server_url = https://rancher.xx.xx.xx.xx.sslip.io + workload_node_ip = yy.yy.yy.yy + ``` + +8. Paste the `rancher_server_url` from the output above into the browser. Log in when prompted (default username is `admin`, use the password set in `rancher_server_admin_password`). +9. ssh to the Rancher Server using the `id_rsa` key generated in `quickstart/azure`. + +#### Result + +Two Kubernetes clusters are deployed into your Azure account, one running Rancher Server and the other ready for experimentation deployments. Please note that while this setup is a great way to explore Rancher functionality, a production setup should follow our high availability setup guidelines. SSH keys for the VMs are auto-generated and stored in the module directory. + +### What's Next? + +Use Rancher to create a deployment. For more information, see [Creating Deployments]({{< baseurl >}}/rancher/v2.6/en/quick-start-guide/workload). + +## Destroying the Environment + +1. From the `quickstart/azure` folder, execute `terraform destroy --auto-approve`. + +2. Wait for confirmation that all resources have been destroyed. diff --git a/content/rancher/v2.6/en/quick-start-guide/deployment/quickstart-manual-setup/_index.md b/content/rancher/v2.6/en/quick-start-guide/deployment/quickstart-manual-setup/_index.md index 0103875d09e..b24baf5a47e 100644 --- a/content/rancher/v2.6/en/quick-start-guide/deployment/quickstart-manual-setup/_index.md +++ b/content/rancher/v2.6/en/quick-start-guide/deployment/quickstart-manual-setup/_index.md @@ -1,116 +1,137 @@ --- -title: Manual Quick Start +title: Helm CLI Quick Start weight: 300 --- -Howdy Partner! This tutorial walks you through: -- Installation of Rancher 2.x -- Creation of your first cluster -- Deployment of an application, Nginx +These instructions capture a quick way to set up a proof-of-concept Rancher installation. -## Quick Start Outline +These instructions assume you have a Linux virtual machine that you will communicate with from your local workstation. Rancher will be installed on the Linux machine. You will need to retrieve the IP address of that machine so that you can access Rancher from your local workstation. Rancher is designed to manage Kubernetes clusters remotely, so any Kubernetes cluster that Rancher manages in the future will also need to be able to reach this IP address. -This Quick Start Guide is divided into different tasks for easier consumption. +We don't recommend installing Rancher locally because it creates a networking problem. Installing Rancher on localhost does not allow Rancher to communicate with downstream Kubernetes clusters, so on localhost you wouldn't be able to test Rancher's cluster provisioning or cluster management functionality. - +Your Linux machine can be anywhere. It could be an Amazon EC2 instance, a Digital Ocean droplet, or an Azure virtual machine, to name a few examples. Other Rancher docs often use 'node' as a generic term for all of these. One possible way to deploy a Linux machine is by setting up an Amazon EC2 instance as shown in [this tutorial]({{}}/rancher/v2.6/en/installation/resources/k8s-tutorials/infrastructure-tutorials/ec2-node/). + +The full installation requirements are [here]({{}}/rancher/v2.6/en/installation/requirements/). -1. [Provision a Linux Host](#1-provision-a-linux-host) +## Install K3s on Linux -1. [Install Rancher](#2-install-rancher) +Install a K3s cluster by running this command on the Linux machine: -1. [Log In](#3-log-in) +``` +curl -sfL https://get.k3s.io | sh -s - server +``` -1. [Create the Cluster](#4-create-the-cluster) +Save the IP of the Linux machine. - -
-### 1. Provision a Linux Host +## Save the kubeconfig to your workstation - Begin creation of a custom cluster by provisioning a Linux host. Your host can be: +The kubeconfig file is important for accessing the Kubernetes cluster. Copy the file at `/etc/rancher/k3s/k3s.yaml` from the Linux machine and save it to your local workstation in the directory `~/.kube/config`. One way to do this is by using the `scp` tool and run this command on your local machine: -- A cloud-host virtual machine (VM) -- An on-prem VM -- A bare-metal server +{{% tabs %}} +{{% tab "Mac and Linux" %}} - >**Note:** - > When using a cloud-hosted virtual machine you need to allow inbound TCP communication to ports 80 and 443. Please see your cloud-host's documentation for information regarding port configuration. - > - > For a full list of port requirements, refer to [Docker Installation]({{}}/rancher/v2.6/en/cluster-provisioning/node-requirements/). +``` +scp root@:/etc/rancher/k3s/k3s.yaml ~/.kube/config +``` - Provision the host according to our [Requirements]({{}}/rancher/v2.6/en/installation/requirements/). +{{% /tab %}} +{{% tab "Windows" %}} -### 2. Install Rancher +By default, "scp" is not a recognized command, so we need to install a module first. -To install Rancher on your host, connect to it and then use a shell to install. +In Windows Powershell: -1. Log in to your Linux host using your preferred shell, such as PuTTy or a remote Terminal connection. +``` +Find-Module Posh-SSH +Install-Module Posh-SSH -2. From your shell, enter the following command: +## Get the remote kubeconfig file +scp root@:/etc/rancher/k3s/k3s.yaml $env:USERPROFILE\.kube\config +``` - ``` - sudo docker run -d --restart=unless-stopped -p 80:80 -p 443:443 --privileged rancher/rancher - ``` +{{% /tab %}} +{{% /tabs %}} -**Result:** Rancher is installed. +## Edit the Rancher server URL in the kubeconfig -### 3. Log In +In the kubeconfig file, the server directive is defined as `localhost`. You will need to change the server directive from `localhost` to `:6443`. The Kubernetes API server will be reached at port 6443, while the Rancher server will be reached at ports 80 and 443. This edit is needed so that when you run Helm or kubectl commands from your local workstation, you will be able to communicate with the Kubernetes cluster that Rancher will be installed on. -Log in to Rancher to begin using the application. After you log in, you'll make some one-time configurations. +{{% tabs %}} +{{% tab "Mac and Linux" %}} -1. Open a web browser and enter the IP address of your host: `https://`. +One way to open the kubeconfig file for editing is to use Vim: - Replace `` with your host IP address. +``` +vi ~/.kube/config +``` -2. When prompted, create a password for the default `admin` account there cowpoke! +Press `i` to put Vim in insert mode. To save your work, press `Esc`. Then press `:wq` and press `Enter`. -3. Set the **Rancher Server URL**. The URL can either be an IP address or a host name. However, each node added to your cluster must be able to connect to this URL.

If you use a hostname in the URL, this hostname must be resolvable by DNS on the nodes you want to add to you cluster. +{{% /tab %}} +{{% tab "Windows" %}} -
+In Windows Powershell, you can use `notepad.exe` for editing the kubeconfig file: -### 4. Create the Cluster +``` +notepad.exe $env:USERPROFILE\.kube\config +``` -Welcome to Rancher! You are now able to create your first Kubernetes cluster. +Once edited, either press `ctrl+s` or go to `File > Save` to save your work. -In this task, you can use the versatile **Custom** option. This option lets you add _any_ Linux host (cloud-hosted VM, on-prem VM, or bare-metal) to be used in a cluster. -1. Click **☰ > Cluster Management**. -1. From the **Clusters** page, click **Create**. -2. Choose **Custom**. +{{% /tab %}} +{{% /tabs %}} -3. Enter a **Cluster Name**. +## Install Rancher with Helm -4. Skip **Member Roles** and **Cluster Options**. We'll tell you about them later. +Then from your local workstation, run the following commands. You will need to have [kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) and [helm.](https://helm.sh/docs/intro/install/) installed. -5. Click **Next**. +``` +helm repo add rancher-latest https://releases.rancher.com/server-charts/latest -6. From **Node Role**, select _all_ the roles: **etcd**, **Control**, and **Worker**. +kubectl create namespace cattle-system -7. **Optional**: Rancher auto-detects the IP addresses used for Rancher communication and cluster communication. You can override these using `Public Address` and `Internal Address` in the **Node Address** section. +kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.7.1/cert-manager.crds.yaml -8. Skip the **Labels** stuff. It's not important for now. +helm repo add jetstack https://charts.jetstack.io -9. Copy the command displayed on screen to your clipboard. +helm repo update -10. Log in to your Linux host using your preferred shell, such as PuTTy or a remote Terminal connection. Run the command copied to your clipboard. +helm install cert-manager jetstack/cert-manager \ + --namespace cert-manager \ + --create-namespace \ + --version v1.7.1 -11. When you finish running the command on your Linux host, click **Done**. +# Windows Powershell +helm install cert-manager jetstack/cert-manager ` + --namespace cert-manager ` + --create-namespace ` + --version v1.7.1 +``` -**Result:** +The final command to install Rancher is below. The command requires a domain name that forwards traffic to the Linux machine. For the sake of simplicity in this tutorial, you can use a fake domain name to create your proof-of-concept. An example of a fake domain name would be `.sslip.io`. -Your cluster is created and assigned a state of **Provisioning**. Rancher is standing up your cluster. +``` +helm install rancher rancher-latest/rancher \ + --namespace cattle-system \ + --set hostname=.sslip.io \ + --set replicas=1 \ + --set bootstrapPassword= -You can access your cluster after its state is updated to **Active**. +# Windows Powershell +helm install rancher rancher-latest/rancher ` + --namespace cattle-system ` + --set hostname=.sslip.io ` + --set replicas=1 ` + --set bootstrapPassword= +``` +``` -**Active** clusters are assigned two Projects: +Now if you navigate to `.sslip.io` in a web browser, you should see the Rancher UI. -- `Default`, containing the `default` namespace -- `System`, containing the `cattle-system`, `ingress-nginx`, `kube-public`, and `kube-system` namespaces +To make these instructions simple, we used a fake domain name and self-signed certificates to do this installation. Therefore, you will probably need to add a security exception to your web browser to see the Rancher UI. Note that for production installs, you would need a high-availability setup with a load balancer, a real domain name and real certificates. -#### Finished +These instructions also left out the full installation requirements and other installation options. If you have any issues with these steps, refer to the full [Helm CLI installation docs.]({{}}/rancher/v2.6/en/installation/install-rancher-on-k8s/) -Congratulations! You have created your first cluster. - -#### What's Next? - -Use Rancher to create a deployment. For more information, see [Creating Deployments]({{}}/rancher/v2.6/en/quick-start-guide/workload). +To launch new Kubernetes clusters with your new Rancher server, you may need to set up cloud credentials in Rancher. For more information, see [Launching Kubernetes clusters with Rancher.]({{}}/rancher/v2.6/en/cluster-provisioning/rke-clusters/) diff --git a/content/rancher/v2.6/en/quick-start-guide/deployment/quickstart-vagrant/_index.md b/content/rancher/v2.6/en/quick-start-guide/deployment/quickstart-vagrant/_index.md index 3f22066b26a..86b3bb420c3 100644 --- a/content/rancher/v2.6/en/quick-start-guide/deployment/quickstart-vagrant/_index.md +++ b/content/rancher/v2.6/en/quick-start-guide/deployment/quickstart-vagrant/_index.md @@ -4,6 +4,8 @@ weight: 200 --- The following steps quickly deploy a Rancher Server with a single node cluster attached. +>**Note:** The intent of these guides is to quickly launch a sandbox that you can use to evaluate Rancher. These guides are not intended for production environments. For comprehensive setup instructions, see [Installation]({{}}/rancher/v2.6/en/installation/). + ## Prerequisites - [Vagrant](https://www.vagrantup.com): Vagrant is required as this is used to provision the machine based on the Vagrantfile. @@ -14,7 +16,7 @@ The following steps quickly deploy a Rancher Server with a single node cluster a - Vagrant will require plugins to create VirtualBox VMs. Install them with the following commands: `vagrant plugin install vagrant-vboxmanage` - + `vagrant plugin install vagrant-vbguest` ## Getting Started @@ -30,7 +32,7 @@ The following steps quickly deploy a Rancher Server with a single node cluster a 4. To initiate the creation of the environment run, `vagrant up --provider=virtualbox`. -5. Once provisioning finishes, go to `https://172.22.101.101` in the browser. The default user/password is `admin/admin`. +5. Once provisioning finishes, go to `https://192.168.56.101` in the browser. The default user/password is `admin/adminPassword`. **Result:** Rancher Server and your Kubernetes cluster is installed on VirtualBox. diff --git a/content/rancher/v2.6/en/quick-start-guide/workload/quickstart-deploy-workload-ingress/_index.md b/content/rancher/v2.6/en/quick-start-guide/workload/quickstart-deploy-workload-ingress/_index.md index fb4ac811514..004950b7fe2 100644 --- a/content/rancher/v2.6/en/quick-start-guide/workload/quickstart-deploy-workload-ingress/_index.md +++ b/content/rancher/v2.6/en/quick-start-guide/workload/quickstart-deploy-workload-ingress/_index.md @@ -13,13 +13,14 @@ You're ready to create your first Kubernetes [workload](https://kubernetes.io/do For this workload, you'll be deploying the application Rancher Hello-World. -1. Click **☰ > Cluster Management**. +1. Click **☰ > Cluster Management**. 1. Go to the cluster that you created and click **Explore**. 1. Click **Workload**. 1. Click **Create**. 1. Click **Deployment**. 1. Enter a **Name** for your workload. 1. From the **Docker Image** field, enter `rancher/hello-world`. This field is case-sensitive. +1. Click **Add Port** and enter `80` in the **Private Container Port** field. Adding a port enables access to the application inside and outside of the cluster. For more information, see [Services]({{}}/rancher/v2.6/en/k8s-in-rancher/workloads/#services). 1. Click **Create**. **Result:** @@ -30,32 +31,33 @@ For this workload, you'll be deploying the application Rancher Hello-World.
### 2. Expose The Application Via An Ingress -Now that the application is up and running it needs to be exposed so that other services can connect. +Now that the application is up and running, it needs to be exposed so that other services can connect. 1. Click **☰ > Cluster Management**. -1. Go to the cluster that you created and click **Explore**. +1. Go to the cluster that you created and click **Explore**. -2. From the main menu of the **Dashboard**, select **Projects**. +1. Click **Service Discovery > Ingresses**. -3. Open the **Default** project. +1. Click **Create.** -4. Click **Resources > Workloads > Load Balancing**. Click on the **Load Balancing** tab. +1. When choosing **Namespace**, ensure it is the same as the one used when you created your deployment. Otherwise, your deployment will not be available when you attempt to select **Target Service**, as in Step 8 below. -5. Click **Add Ingress**. +1. Enter a **Name**, such as **hello**. -6. Enter a name i.e. **hello**. +1. Specify your **Path**, such as `/hello`. -7. In the **Target** field, drop down the list and choose the name that you set for your service. +1. In the **Target Service** field, drop down the list and choose the name that you set for your service. -8. Enter `80` in the **Port** field. +1. In the **Port** field, drop down the list and select `80`. -9. Leave everything else as default and click **Save**. +1. Click **Create** at the bottom right. + +**Result:** The application is assigned a `sslip.io` address and exposed. It may take a minute or two to populate. -**Result:** The application is assigned a `xip.io` address and exposed. It may take a minute or two to populate. ### View Your Application -From the **Load Balancing** page, click the target link, which will look something like `hello.default.xxx.xxx.xxx.xxx.xip.io > hello-world`. +From the **Deployments** page, find the **Endpoints** column for your deployment and click on an endpoint. The endpoints available will depend on how you configured the port you added to your deployment. For endpoints where you do not see a randomly assigned port, append the path you specified when creating the ingress to the IP address. For example, if your endpoint looks like `xxx.xxx.xxx.xxx` or `https://xxx.xxx.xxx.xxx` change it to `xxx.xxx.xxx.xxx/hello` or `https://xxx.xxx.xxx.xxx/hello`. Your application will open in a separate window. diff --git a/content/rancher/v2.6/en/security/_index.md b/content/rancher/v2.6/en/security/_index.md index 49518e50f01..8b782b613c5 100644 --- a/content/rancher/v2.6/en/security/_index.md +++ b/content/rancher/v2.6/en/security/_index.md @@ -11,7 +11,7 @@ weight: 20

Reporting process

-

Please submit possible security issues by emailing security@rancher.com

+

Please submit possible security issues by emailing security-rancher@suse.com .

Announcements

@@ -20,24 +20,32 @@ weight: 20 -Security is at the heart of all Rancher features. From integrating with all the popular authentication tools and services, to an enterprise grade [RBAC capability,]({{}}/rancher/v2.6/en/admin-settings/rbac) Rancher makes your Kubernetes clusters even more secure. +Security is at the heart of all Rancher features. From integrating with all the popular authentication tools and services, to an enterprise grade [RBAC capability]({{}}/rancher/v2.6/en/admin-settings/rbac), Rancher makes your Kubernetes clusters even more secure. -On this page, we provide security-related documentation along with resources to help you secure your Rancher installation and your downstream Kubernetes clusters: +On this page, we provide security related documentation along with resources to help you secure your Rancher installation and your downstream Kubernetes clusters: +- [NeuVector Integration with Rancher](#neuvector-integration-with-rancher) - [Running a CIS security scan on a Kubernetes cluster](#running-a-cis-security-scan-on-a-kubernetes-cluster) - [SELinux RPM](#selinux-rpm) - [Guide to hardening Rancher installations](#rancher-hardening-guide) - [The CIS Benchmark and self-assessment](#the-cis-benchmark-and-self-assessment) - [Third-party penetration test reports](#third-party-penetration-test-reports) -- [Rancher CVEs and resolutions](#rancher-cves-and-resolutions) +- [Rancher Security Advisories and CVEs](#rancher-security-advisories-and-cves) +- [Kubernetes Security Best Practices](#kubernetes-security-best-practices) + +### NeuVector Integration with Rancher + +_New in v2.6.5_ + +NeuVector is an open-source, container-focused security application that is now integrated into Rancher. NeuVector provides production security, DevOps vulnerability protection, and a container firewall, et al. Please see the [Rancher docs]({{}}/rancher/v2.6/en/neuvector-integration) and the [NeuVector docs](https://open-docs.neuvector.com/) for more information. ### Running a CIS Security Scan on a Kubernetes Cluster -Rancher leverages [kube-bench](https://github.com/aquasecurity/kube-bench) to run a security scan to check whether Kubernetes is deployed according to security best practices as defined in the CIS (Center for Internet Security) Kubernetes Benchmark. +Rancher leverages [kube-bench](https://github.com/aquasecurity/kube-bench) to run a security scan to check whether Kubernetes is deployed according to security best practices as defined in the [CIS](https://www.cisecurity.org/cis-benchmarks/) (Center for Internet Security) Kubernetes Benchmark. The CIS Kubernetes Benchmark is a reference document that can be used to establish a secure configuration baseline for Kubernetes. -The Center for Internet Security (CIS) is a 501(c\)(3) non-profit organization, formed in October 2000, with a mission to "identify, develop, validate, promote, and sustain best practice solutions for cyber defense and build and lead communities to enable an environment of trust in cyberspace." +The Center for Internet Security (CIS) is a 501(c\)(3) non-profit organization, formed in October 2000, with a mission to "identify, develop, validate, promote, and sustain best practice solutions for cyber defense and build and lead communities to enable an environment of trust in cyberspace". CIS Benchmarks are best practices for the secure configuration of a target system. CIS Benchmarks are developed through the generous volunteer efforts of subject matter experts, technology vendors, public and private community members, and the CIS Benchmark Development team. @@ -45,13 +53,13 @@ The Benchmark provides recommendations of two types: Automated and Manual. We ru When Rancher runs a CIS security scan on a cluster, it generates a report showing the results of each test, including a summary with the number of passed, skipped and failed tests. The report also includes remediation steps for any failed tests. -For details, refer to the section on [security scans.]({{}}/rancher/v2.6/en/cis-scans) +For details, refer to the section on [security scans]({{}}/rancher/v2.6/en/cis-scans). ### SELinux RPM [Security-Enhanced Linux (SELinux)](https://en.wikipedia.org/wiki/Security-Enhanced_Linux) is a security enhancement to Linux. After being historically used by government agencies, SELinux is now industry standard and is enabled by default on CentOS 7 and 8. -We provide two RPMs (Red Hat packages) that enable Rancher products to function properly on SELinux-enforcing hosts: `rancher-selinux` and `rke2-selinux`. For details, see [this page.]({{}}/rancher/v2.6/en/security/selinux) +We provide two RPMs (Red Hat packages) that enable Rancher products to function properly on SELinux-enforcing hosts: `rancher-selinux` and `rke2-selinux`. For details, see [this page]({{}}/rancher/v2.6/en/security/selinux). ### Rancher Hardening Guide @@ -77,9 +85,13 @@ Rancher periodically hires third parties to perform security audits and penetrat Results: -- [Cure53 Pen Test - 7/2019](https://releases.rancher.com/documents/security/pen-tests/2019/RAN-01-cure53-report.final.pdf) -- [Untamed Theory Pen Test- 3/2019](https://releases.rancher.com/documents/security/pen-tests/2019/UntamedTheory-Rancher_SecurityAssessment-20190712_v5.pdf) +- [Cure53 Pen Test - July 2019](https://releases.rancher.com/documents/security/pen-tests/2019/RAN-01-cure53-report.final.pdf) +- [Untamed Theory Pen Test- March 2019](https://releases.rancher.com/documents/security/pen-tests/2019/UntamedTheory-Rancher_SecurityAssessment-20190712_v5.pdf) -### Rancher CVEs and Resolutions +### Rancher Security Advisories and CVEs Rancher is committed to informing the community of security issues in our products. For the list of CVEs (Common Vulnerabilities and Exposures) for issues we have resolved, refer to [this page.](./cve) + +### Kubernetes Security Best Practices + +For recommendations on securing your Kubernetes cluster, refer to the [Kubernetes Security Best Practices](./best-practices) guide. diff --git a/content/rancher/v2.6/en/security/best-practices/_index.md b/content/rancher/v2.6/en/security/best-practices/_index.md new file mode 100644 index 00000000000..4dc70b3d510 --- /dev/null +++ b/content/rancher/v2.6/en/security/best-practices/_index.md @@ -0,0 +1,12 @@ +--- +title: Kubernetes Security Best Practices +weight: 5 +--- + +### Restricting cloud metadata API access + +Cloud providers such as AWS, Azure, DigitalOcean or GCP often expose metadata services locally to instances. By default, this endpoint is accessible by pods running on a cloud instance, including pods in hosted Kubernetes providers such as EKS, AKS, DigitalOcean Kubernetes or GKE, and can contain cloud credentials for that node, provisioning data such as kubelet credentials, or other sensitive data. To mitigate this risk when running on a cloud platform, follow the [Kubernetes security recommendations](https://kubernetes.io/docs/tasks/administer-cluster/securing-a-cluster/#restricting-cloud-metadata-api-access): limit permissions given to instance credentials, use network policies to restrict pod access to the metadata API, and avoid using provisioning data to deliver secrets. + +It is advised to consult your cloud provider's security best practices for further recommendations and specific details on how to restrict access to cloud instance metadata API. + +Further references: MITRE ATT&CK knowledge base on - [Unsecured Credentials: Cloud Instance Metadata API](https://attack.mitre.org/techniques/T1552/005/). diff --git a/content/rancher/v2.6/en/security/cve/_index.md b/content/rancher/v2.6/en/security/cve/_index.md index b97cf1a59c5..b353c508096 100644 --- a/content/rancher/v2.6/en/security/cve/_index.md +++ b/content/rancher/v2.6/en/security/cve/_index.md @@ -1,12 +1,19 @@ --- -title: Rancher CVEs and Resolutions +title: Security Advisories and CVEs weight: 300 --- -Rancher is committed to informing the community of security issues in our products. Rancher will publish CVEs (Common Vulnerabilities and Exposures) for issues we have resolved. +Rancher is committed to informing the community of security issues in our products. Rancher will publish security advisories and CVEs (Common Vulnerabilities and Exposures) for issues we have resolved. New security advisories are also published in Rancher's GitHub [security page](https://github.com/rancher/rancher/security/advisories). | ID | Description | Date | Resolution | |----|-------------|------|------------| +| [CVE-2021-36784](https://github.com/rancher/rancher/security/advisories/GHSA-jwvr-vv7p-gpwq) | A vulnerability was discovered in Rancher versions from 2.5.0 up to and including 2.5.12 and from 2.6.0 up to and including 2.6.3 which allows users who have create or update permissions on [Global Roles](https://rancher.com/docs/rancher/v2.6/en/admin-settings/rbac/) to escalate their permissions, or those of another user, to admin-level permissions. Global Roles grant users Rancher-wide permissions, such as the ability to create clusters. In the identified versions of Rancher, when users are given permission to edit or create Global Roles, they are not restricted to only granting permissions which they already posses. This vulnerability affects customers who utilize non-admin users that are able to create or edit Global Roles. The most common use case for this scenario is the `restricted-admin` role. | 14 Apr 2022 | [Rancher v2.6.4](https://github.com/rancher/rancher/releases/tag/v2.6.4) and [Rancher v2.5.13](https://github.com/rancher/rancher/releases/tag/v2.5.13) | +| [CVE-2021-4200](https://github.com/rancher/rancher/security/advisories/GHSA-hx8w-ghh8-r4xf) | This vulnerability only affects customers using the `restricted-admin` role in Rancher. A vulnerability was discovered in Rancher versions from 2.5.0 up to and including 2.5.12 and from 2.6.0 up to and including 2.6.3 where the `global-data` role in `cattle-global-data` namespace grants write access to the Catalogs. Since each user with any level of catalog access was bound to the `global-data` role, this grants write access to templates (`CatalogTemplates`) and template versions (`CatalogTemplateVersions`) for any user with any level of catalog access. New users created in Rancher are by default assigned to the `user` role (standard user), which is not designed to grant write catalog access. This vulnerability effectively elevates the privilege of any user to write access for the catalog template and catalog template version resources. | 14 Apr 2022 | [Rancher v2.6.4](https://github.com/rancher/rancher/releases/tag/v2.6.4) and [Rancher v2.5.13](https://github.com/rancher/rancher/releases/tag/v2.5.13) | +| [GHSA-wm2r-rp98-8pmh](https://github.com/rancher/rancher/security/advisories/GHSA-wm2r-rp98-8pmh) | This vulnerability only affects customers using [Fleet](https://rancher.com/docs/rancher/v2.6/en/deploy-across-clusters/fleet/) for continuous delivery with authenticated Git and/or Helm repositories. An issue was discovered in `go-getter` library in versions prior to [`v1.5.11`](https://github.com/hashicorp/go-getter/releases/tag/v1.5.11) that exposes SSH private keys in base64 format due to a failure in redacting such information from error messages. The vulnerable version of this library is used in Rancher through Fleet in versions of Fleet prior to [`v0.3.9`](https://github.com/rancher/fleet/releases/tag/v0.3.9). This issue affects Rancher versions 2.5.0 up to and including 2.5.12 and from 2.6.0 up to and including 2.6.3. The issue was found and reported by Dagan Henderson from Raft Engineering. | 14 Apr 2022 | [Rancher v2.6.4](https://github.com/rancher/rancher/releases/tag/v2.6.4) and [Rancher v2.5.13](https://github.com/rancher/rancher/releases/tag/v2.5.13) | +| [CVE-2021-36778](https://github.com/rancher/rancher/security/advisories/GHSA-4fc7-hc63-7fjg) | A vulnerability was discovered in Rancher versions from 2.5.0 up to and including 2.5.11 and from 2.6.0 up to and including 2.6.2, where an insufficient check of the same-origin policy when downloading Helm charts from a configured private repository can lead to exposure of the repository credentials to a third-party provider. This issue only happens when the user configures access credentials to a private repository in Rancher inside `Apps & Marketplace > Repositories`. The issue was found and reported by Martin Andreas Ullrich. | 14 Apr 2022 | [Rancher v2.6.3](https://github.com/rancher/rancher/releases/tag/v2.6.3) and [Rancher v2.5.12](https://github.com/rancher/rancher/releases/tag/v2.5.12) | +| [GHSA-hwm2-4ph6-w6m5](https://github.com/rancher/rancher/security/advisories/GHSA-hwm2-4ph6-w6m5) | A vulnerability was discovered in versions of Rancher starting 2.0 up to and including 2.6.3. The `restricted` pod security policy (PSP) provided in Rancher deviated from the upstream `restricted` policy provided in Kubernetes on account of which Rancher's PSP had `runAsUser` set to `runAsAny`, while upstream had `runAsUser` set to `MustRunAsNonRoot`. This allowed containers to run as any user, including a privileged user (`root`), even when Rancher's `restricted` policy was enforced on a project or at the cluster level. | 31 Mar 2022 | [Rancher v2.6.4](https://github.com/rancher/rancher/releases/tag/v2.6.4) | +| [CVE-2021-36775](https://github.com/rancher/rancher/security/advisories/GHSA-28g7-896h-695v) | A vulnerability was discovered in Rancher versions up to and including 2.4.17, 2.5.11 and 2.6.2. After removing a `Project Role` associated with a group from the project, the bindings that granted access to cluster-scoped resources for those subjects were not deleted. This was due to an incomplete authorization logic check. A user who was a member of the affected group with authenticated access to Rancher could exploit this vulnerability to access resources they shouldn't have had access to. The exposure level would depend on the original permission level granted to the affected project role. This vulnerability only affected customers using group based authentication in Rancher. | 31 Mar 2022 | [Rancher v2.6.3](https://github.com/rancher/rancher/releases/tag/v2.6.3), [Rancher v2.5.12](https://github.com/rancher/rancher/releases/tag/v2.5.12) and [Rancher v2.4.18](https://github.com/rancher/rancher/releases/tag/v2.4.18) | +| [CVE-2021-36776](https://github.com/rancher/rancher/security/advisories/GHSA-gvh9-xgrq-r8hw) | A vulnerability was discovered in Rancher versions starting 2.5.0 up to and including 2.5.9, that allowed an authenticated user to impersonate any user on a cluster through an API proxy, without requiring knowledge of the impersonated user's credentials. This was due to the API proxy not dropping the impersonation header before sending the request to the Kubernetes API. A malicious user with authenticated access to Rancher could use this to impersonate another user with administrator access in Rancher, thereby gaining administrator level access to the cluster. | 31 Mar 2022 | [Rancher v2.6.0](https://github.com/rancher/rancher/releases/tag/v2.6.0) and [Rancher v2.5.10](https://github.com/rancher/rancher/releases/tag/v2.5.10) | | [CVE-2021-25318](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-25318) | A vulnerability was discovered in Rancher versions 2.0 through the aforementioned fixed versions, where users were granted access to resources regardless of the resource's API group. For example, Rancher should have allowed users access to `apps.catalog.cattle.io`, but instead incorrectly gave access to `apps.*`. Resources affected in the **Downstream clusters** and **Rancher management cluster** can be found [here](https://github.com/rancher/rancher/security/advisories/GHSA-f9xf-jq4j-vqw4). There is not a direct mitigation besides upgrading to the patched Rancher versions. | 14 Jul 2021 | [Rancher v2.5.9](https://github.com/rancher/rancher/releases/tag/v2.5.9) and [Rancher v2.4.16](https://github.com/rancher/rancher/releases/tag/v2.4.16) | | [CVE-2021-31999](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-31999) | A vulnerability was discovered in Rancher 2.0.0 through the aforementioned patched versions, where a malicious Rancher user could craft an API request directed at the proxy for the Kubernetes API of a managed cluster to gain access to information they do not have access to. This is done by passing the "Impersonate-User" or "Impersonate-Group" header in the Connection header, which is then correctly removed by the proxy. At this point, instead of impersonating the user and their permissions, the request will act as if it was from the Rancher management server and incorrectly return the information. The vulnerability is limited to valid Rancher users with some level of permissions on the cluster. There is not a direct mitigation besides upgrading to the patched Rancher versions. | 14 Jul 2021 | [Rancher v2.5.9](https://github.com/rancher/rancher/releases/tag/v2.5.9) and [Rancher v2.4.16](https://github.com/rancher/rancher/releases/tag/v2.4.16) | | [CVE-2021-25320](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-25320) | A vulnerability was discovered in Rancher 2.2.0 through the aforementioned patched versions, where cloud credentials weren't being properly validated through the Rancher API. Specifically through a proxy designed to communicate with cloud providers. Any Rancher user that was logged-in and aware of a cloud-credential ID that was valid for a given cloud provider, could call that cloud provider's API through the proxy API, and the cloud-credential would be attached. The exploit is limited to valid Rancher users. There is not a direct mitigation outside of upgrading to the patched Rancher versions. | 14 Jul 2021 | [Rancher v2.5.9](https://github.com/rancher/rancher/releases/tag/v2.5.9) and [Rancher v2.4.16](https://github.com/rancher/rancher/releases/tag/v2.4.16) | @@ -18,4 +25,4 @@ Rancher is committed to informing the community of security issues in our produc | [CVE-2019-12274](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-12274) | Nodes using the built-in node drivers using a file path option allows the machine to read arbitrary files including sensitive ones from inside the Rancher server container. | 5 Jun 2019 | [Rancher v2.2.4](https://github.com/rancher/rancher/releases/tag/v2.2.4), [Rancher v2.1.10](https://github.com/rancher/rancher/releases/tag/v2.1.10) and [Rancher v2.0.15](https://github.com/rancher/rancher/releases/tag/v2.0.15) | | [CVE-2019-11202](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-11202) | The default admin, that is shipped with Rancher, will be re-created upon restart of Rancher despite being explicitly deleted. | 16 Apr 2019 | [Rancher v2.2.2](https://github.com/rancher/rancher/releases/tag/v2.2.2), [Rancher v2.1.9](https://github.com/rancher/rancher/releases/tag/v2.1.9) and [Rancher v2.0.14](https://github.com/rancher/rancher/releases/tag/v2.0.14) | | [CVE-2019-6287](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-6287) | Project members continue to get access to namespaces from projects that they were removed from if they were added to more than one project. | 29 Jan 2019 | [Rancher v2.1.6](https://github.com/rancher/rancher/releases/tag/v2.1.6) and [Rancher v2.0.11](https://github.com/rancher/rancher/releases/tag/v2.0.11) | -| [CVE-2018-20321](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-20321) | Any project member with access to the `default` namespace can mount the `netes-default` service account in a pod and then use that pod to execute administrative privileged commands against the Kubernetes cluster. | 29 Jan 2019 | [Rancher v2.1.6](https://github.com/rancher/rancher/releases/tag/v2.1.6) and [Rancher v2.0.11](https://github.com/rancher/rancher/releases/tag/v2.0.11) - Rolling back from these versions or greater have specific [instructions]({{}}/rancher/v2.6/en/installation/install-rancher-on-k8s/rollbacks). | \ No newline at end of file +| [CVE-2018-20321](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-20321) | Any project member with access to the `default` namespace can mount the `netes-default` service account in a pod and then use that pod to execute administrative privileged commands against the Kubernetes cluster. | 29 Jan 2019 | [Rancher v2.1.6](https://github.com/rancher/rancher/releases/tag/v2.1.6) and [Rancher v2.0.11](https://github.com/rancher/rancher/releases/tag/v2.0.11) - Rolling back from these versions or greater have specific [instructions]({{}}/rancher/v2.6/en/installation/install-rancher-on-k8s/rollbacks). | diff --git a/content/rancher/v2.6/en/security/hardening-guides/1.6-benchmark-2.6/Rancher_v2-6_CIS_v1-6_Benchmark_Assessment.pdf b/content/rancher/v2.6/en/security/hardening-guides/1.6-benchmark-2.6/Rancher_v2-6_CIS_v1-6_Benchmark_Assessment.pdf new file mode 100644 index 00000000000..f08210f51e6 --- /dev/null +++ b/content/rancher/v2.6/en/security/hardening-guides/1.6-benchmark-2.6/Rancher_v2-6_CIS_v1-6_Benchmark_Assessment.pdf @@ -0,0 +1,14832 @@ +%PDF-1.7 +%🖤 +1 0 obj +<< +/Type /Pages +/Kids [ 6 0 R 8 0 R 38 0 R 76 0 R 114 0 R 152 0 R 186 0 R 222 0 R 258 0 R 292 0 R 304 0 R 309 0 R 313 0 R 315 0 R 318 0 R 323 0 R 325 0 R 349 0 R 374 0 R 407 0 R 440 0 R 464 0 R 503 0 R 508 0 R 511 0 R 514 0 R 516 0 R 519 0 R 521 0 R 524 0 R 527 0 R 529 0 R 532 0 R 534 0 R 537 0 R 540 0 R 542 0 R 545 0 R 548 0 R 550 0 R 553 0 R 555 0 R 559 0 R 562 0 R 565 0 R 567 0 R 570 0 R 572 0 R 575 0 R 578 0 R 580 0 R 583 0 R 586 0 R 588 0 R 591 0 R 593 0 R 596 0 R 599 0 R 601 0 R 604 0 R 607 0 R 609 0 R 612 0 R 614 0 R 617 0 R 620 0 R 622 0 R 625 0 R 627 0 R 630 0 R 633 0 R 635 0 R 638 0 R 641 0 R 643 0 R 660 0 R 674 0 R 677 0 R 680 0 R 683 0 R 686 0 R 689 0 R 692 0 R 694 0 R 698 0 R 700 0 R 703 0 R 705 0 R 708 0 R 710 0 R 713 0 R 715 0 R 717 0 R 720 0 R 722 0 R 725 0 R 727 0 R 730 0 R 732 0 R 734 0 R 737 0 R 739 0 R 741 0 R 743 0 R 746 0 R 748 0 R 751 0 R 755 0 R 764 0 R 772 0 R 774 0 R 779 0 R 784 0 R 790 0 R 794 0 R 798 0 R 804 0 R 808 0 R 812 0 R 815 0 R 817 0 R 842 0 R 853 0 R 856 0 R 860 0 R 863 0 R 880 0 R 887 0 R 889 0 R 891 0 R 893 0 R ] +/Count 131 +>> +endobj +2 0 obj +<< +/Title (CIS v1.6 Benchmark - Self-Assessment Guide - Rancher v2.6) +/Creator (pandoc) +/Producer (WeasyPrint 54.2) +>> +endobj +3 0 obj +<< +/Type /Catalog +/Pages 1 0 R +/Outlines 1055 0 R +/Names << +/Dests << +/Names [ (TOC) [ 8 0 R /XYZ 77.509843 756.850394 0 ] (api-server) [ 508 0 R /XYZ 41.740157 784.430079 0 ] (apply-security-context-to-your-pods-and-containers-manual) [ 891 0 R /XYZ 84.259843 324.051496 0 ] (authentication-and-authorization) [ 741 0 R /XYZ 41.740157 784.430079 0 ] (cb1) [ 315 0 R /XYZ 84.259843 207.170079 0 ] (cb1-1) [ 315 0 R /XYZ 101.433071 205.473057 0 ] (cb10) [ 349 0 R /XYZ 84.259843 342.035606 0 ] (cb10-1) [ 349 0 R /XYZ 101.433071 340.338585 0 ] (cb101) [ 622 0 R /XYZ 84.259843 504.320079 0 ] (cb101-1) [ 622 0 R /XYZ 101.433071 502.623057 0 ] (cb104) [ 627 0 R /XYZ 84.259843 772.370079 0 ] (cb104-1) [ 627 0 R /XYZ 101.433071 770.673057 0 ] (cb107) [ 630 0 R /XYZ 84.259843 313.070079 0 ] (cb107-1) [ 630 0 R /XYZ 101.433071 311.373057 0 ] (cb110) [ 635 0 R /XYZ 84.259843 560.570079 0 ] (cb110-1) [ 635 0 R /XYZ 101.433071 558.873057 0 ] (cb113) [ 638 0 R /XYZ 84.259843 110.570079 0 ] (cb113-1) [ 638 0 R /XYZ 101.433071 108.873057 0 ] (cb116) [ 643 0 R /XYZ 84.259843 418.070079 0 ] (cb116-1) [ 643 0 R /XYZ 101.433071 416.373057 0 ] (cb116-10) [ 643 0 R /XYZ 99.933071 189.008799 0 ] (cb116-11) [ 643 0 R /XYZ 99.933071 170.061778 0 ] (cb116-12) [ 643 0 R /XYZ 99.933071 151.114756 0 ] (cb116-13) [ 643 0 R /XYZ 99.933071 132.167735 0 ] (cb116-14) [ 643 0 R /XYZ 99.933071 113.220714 0 ] (cb116-15) [ 643 0 R /XYZ 99.933071 94.273692 0 ] (cb116-16) [ 660 0 R /XYZ 99.933071 799.173057 0 ] (cb116-17) [ 660 0 R /XYZ 99.933071 780.226036 0 ] (cb116-18) [ 660 0 R /XYZ 99.933071 761.279014 0 ] (cb116-19) [ 660 0 R /XYZ 99.933071 742.331993 0 ] (cb116-2) [ 643 0 R /XYZ 99.933071 397.426036 0 ] (cb116-20) [ 660 0 R /XYZ 99.933071 723.384971 0 ] (cb116-21) [ 660 0 R /XYZ 99.933071 704.437950 0 ] (cb116-22) [ 660 0 R /XYZ 99.933071 685.490928 0 ] (cb116-23) [ 660 0 R /XYZ 99.933071 666.543907 0 ] (cb116-24) [ 660 0 R /XYZ 99.933071 647.596885 0 ] (cb116-25) [ 660 0 R /XYZ 99.933071 628.649864 0 ] (cb116-3) [ 643 0 R /XYZ 99.933071 378.479014 0 ] (cb116-4) [ 643 0 R /XYZ 99.933071 340.584971 0 ] (cb116-5) [ 643 0 R /XYZ 99.933071 321.637950 0 ] (cb116-6) [ 643 0 R /XYZ 99.933071 302.690928 0 ] (cb116-7) [ 643 0 R /XYZ 99.933071 283.743907 0 ] (cb116-8) [ 643 0 R /XYZ 99.933071 264.796885 0 ] (cb116-9) [ 643 0 R /XYZ 99.933071 226.902842 0 ] (cb117) [ 660 0 R /XYZ 84.259843 569.399864 0 ] (cb117-1) [ 660 0 R /XYZ 101.433071 567.702842 0 ] (cb120) [ 660 0 R /XYZ 84.259843 196.152842 0 ] (cb120-1) [ 660 0 R /XYZ 101.433071 194.455821 0 ] (cb121) [ 674 0 R /XYZ 84.259843 550.251496 0 ] (cb121-1) [ 674 0 R /XYZ 101.433071 548.554475 0 ] (cb124) [ 677 0 R /XYZ 84.259843 571.820079 0 ] (cb124-1) [ 677 0 R /XYZ 101.433071 570.123057 0 ] (cb127) [ 680 0 R /XYZ 84.259843 575.570079 0 ] (cb127-1) [ 680 0 R /XYZ 101.433071 573.873057 0 ] (cb13) [ 374 0 R /XYZ 84.259843 711.620079 0 ] (cb13-1) [ 374 0 R /XYZ 101.433071 709.923057 0 ] (cb13-10) [ 374 0 R /XYZ 99.933071 501.505821 0 ] (cb13-11) [ 374 0 R /XYZ 99.933071 482.558799 0 ] (cb13-12) [ 374 0 R /XYZ 99.933071 463.611778 0 ] (cb13-13) [ 374 0 R /XYZ 99.933071 444.664756 0 ] (cb13-14) [ 374 0 R /XYZ 99.933071 425.717735 0 ] (cb13-15) [ 374 0 R /XYZ 99.933071 406.770714 0 ] (cb13-16) [ 374 0 R /XYZ 99.933071 387.823692 0 ] (cb13-17) [ 374 0 R /XYZ 99.933071 368.876671 0 ] (cb13-18) [ 374 0 R /XYZ 99.933071 349.929649 0 ] (cb13-19) [ 374 0 R /XYZ 99.933071 330.982628 0 ] (cb13-2) [ 374 0 R /XYZ 99.933071 690.976036 0 ] (cb13-20) [ 374 0 R /XYZ 99.933071 312.035606 0 ] (cb13-21) [ 374 0 R /XYZ 99.933071 293.088585 0 ] (cb13-22) [ 374 0 R /XYZ 99.933071 274.141563 0 ] (cb13-23) [ 374 0 R /XYZ 99.933071 255.194542 0 ] (cb13-24) [ 374 0 R /XYZ 99.933071 236.247520 0 ] (cb13-25) [ 374 0 R /XYZ 99.933071 217.300499 0 ] (cb13-26) [ 374 0 R /XYZ 99.933071 198.353477 0 ] (cb13-27) [ 374 0 R /XYZ 99.933071 179.406456 0 ] (cb13-28) [ 374 0 R /XYZ 99.933071 160.459434 0 ] (cb13-29) [ 374 0 R /XYZ 99.933071 141.512413 0 ] (cb13-3) [ 374 0 R /XYZ 99.933071 672.029014 0 ] (cb13-30) [ 374 0 R /XYZ 99.933071 122.565391 0 ] (cb13-31) [ 374 0 R /XYZ 99.933071 103.618370 0 ] (cb13-32) [ 407 0 R /XYZ 99.933071 799.173057 0 ] (cb13-33) [ 407 0 R /XYZ 99.933071 780.226036 0 ] (cb13-34) [ 407 0 R /XYZ 99.933071 761.279014 0 ] (cb13-35) [ 407 0 R /XYZ 99.933071 742.331993 0 ] (cb13-36) [ 407 0 R /XYZ 99.933071 723.384971 0 ] (cb13-37) [ 407 0 R /XYZ 99.933071 704.437950 0 ] (cb13-38) [ 407 0 R /XYZ 99.933071 685.490928 0 ] (cb13-39) [ 407 0 R /XYZ 99.933071 666.543907 0 ] (cb13-4) [ 374 0 R /XYZ 99.933071 634.134971 0 ] (cb13-40) [ 407 0 R /XYZ 99.933071 647.596885 0 ] (cb13-41) [ 407 0 R /XYZ 99.933071 628.649864 0 ] (cb13-42) [ 407 0 R /XYZ 99.933071 609.702842 0 ] (cb13-43) [ 407 0 R /XYZ 99.933071 590.755821 0 ] (cb13-44) [ 407 0 R /XYZ 99.933071 571.808799 0 ] (cb13-45) [ 407 0 R /XYZ 99.933071 552.861778 0 ] (cb13-46) [ 407 0 R /XYZ 99.933071 533.914756 0 ] (cb13-47) [ 407 0 R /XYZ 99.933071 514.967735 0 ] (cb13-48) [ 407 0 R /XYZ 99.933071 496.020714 0 ] (cb13-49) [ 407 0 R /XYZ 99.933071 477.073692 0 ] (cb13-5) [ 374 0 R /XYZ 99.933071 596.240928 0 ] (cb13-50) [ 407 0 R /XYZ 99.933071 458.126671 0 ] (cb13-51) [ 407 0 R /XYZ 99.933071 439.179649 0 ] (cb13-52) [ 407 0 R /XYZ 99.933071 420.232628 0 ] (cb13-53) [ 407 0 R /XYZ 99.933071 382.338585 0 ] (cb13-54) [ 407 0 R /XYZ 99.933071 363.391563 0 ] (cb13-55) [ 407 0 R /XYZ 99.933071 344.444542 0 ] (cb13-56) [ 407 0 R /XYZ 99.933071 325.497520 0 ] (cb13-57) [ 407 0 R /XYZ 99.933071 306.550499 0 ] (cb13-58) [ 407 0 R /XYZ 99.933071 287.603477 0 ] (cb13-59) [ 407 0 R /XYZ 99.933071 268.656456 0 ] (cb13-6) [ 374 0 R /XYZ 99.933071 577.293907 0 ] (cb13-60) [ 407 0 R /XYZ 99.933071 249.709434 0 ] (cb13-61) [ 407 0 R /XYZ 99.933071 230.762413 0 ] (cb13-7) [ 374 0 R /XYZ 99.933071 558.346885 0 ] (cb13-8) [ 374 0 R /XYZ 99.933071 539.399864 0 ] (cb13-9) [ 374 0 R /XYZ 99.933071 520.452842 0 ] (cb130) [ 683 0 R /XYZ 84.259843 560.570079 0 ] (cb130-1) [ 683 0 R /XYZ 101.433071 558.873057 0 ] (cb133) [ 686 0 R /XYZ 84.259843 556.820079 0 ] (cb133-1) [ 686 0 R /XYZ 101.433071 555.123057 0 ] (cb136) [ 689 0 R /XYZ 84.259843 369.020079 0 ] (cb136-1) [ 689 0 R /XYZ 101.433071 367.323057 0 ] (cb139) [ 694 0 R /XYZ 84.259843 565.251496 0 ] (cb139-1) [ 694 0 R /XYZ 101.433071 563.554475 0 ] (cb14) [ 407 0 R /XYZ 84.259843 171.512413 0 ] (cb14-1) [ 407 0 R /XYZ 101.433071 169.815391 0 ] (cb142) [ 694 0 R /XYZ 84.259843 102.004475 0 ] (cb142-1) [ 694 0 R /XYZ 101.433071 100.307453 0 ] (cb145) [ 700 0 R /XYZ 84.259843 514.251496 0 ] (cb145-1) [ 700 0 R /XYZ 101.433071 512.554475 0 ] (cb148) [ 705 0 R /XYZ 84.259843 309.320079 0 ] (cb148-1) [ 705 0 R /XYZ 101.433071 307.623057 0 ] (cb151) [ 710 0 R /XYZ 84.259843 103.070079 0 ] (cb151-1) [ 710 0 R /XYZ 101.433071 101.373057 0 ] (cb154) [ 717 0 R /XYZ 84.259843 613.970079 0 ] (cb154-1) [ 717 0 R /XYZ 101.433071 612.273057 0 ] (cb157) [ 722 0 R /XYZ 84.259843 403.070079 0 ] (cb157-1) [ 722 0 R /XYZ 101.433071 401.373057 0 ] (cb160) [ 727 0 R /XYZ 84.259843 178.070079 0 ] (cb160-1) [ 727 0 R /XYZ 101.433071 176.373057 0 ] (cb163) [ 734 0 R /XYZ 84.259843 628.970079 0 ] (cb163-1) [ 734 0 R /XYZ 101.433071 627.273057 0 ] (cb166) [ 743 0 R /XYZ 84.259843 595.251496 0 ] (cb166-1) [ 743 0 R /XYZ 101.433071 593.554475 0 ] (cb169) [ 748 0 R /XYZ 84.259843 267.651496 0 ] (cb169-1) [ 748 0 R /XYZ 101.433071 265.954475 0 ] (cb17) [ 440 0 R /XYZ 84.259843 543.320079 0 ] (cb17-1) [ 440 0 R /XYZ 101.433071 541.623057 0 ] (cb17-10) [ 440 0 R /XYZ 99.933071 333.205821 0 ] (cb17-11) [ 440 0 R /XYZ 99.933071 314.258799 0 ] (cb17-12) [ 440 0 R /XYZ 99.933071 295.311778 0 ] (cb17-13) [ 440 0 R /XYZ 99.933071 276.364756 0 ] (cb17-14) [ 440 0 R /XYZ 99.933071 257.417735 0 ] (cb17-15) [ 440 0 R /XYZ 99.933071 238.470714 0 ] (cb17-16) [ 440 0 R /XYZ 99.933071 219.523692 0 ] (cb17-17) [ 440 0 R /XYZ 99.933071 200.576671 0 ] (cb17-18) [ 440 0 R /XYZ 99.933071 181.629649 0 ] (cb17-19) [ 440 0 R /XYZ 99.933071 162.682628 0 ] (cb17-2) [ 440 0 R /XYZ 99.933071 522.676036 0 ] (cb17-20) [ 440 0 R /XYZ 99.933071 143.735606 0 ] (cb17-21) [ 440 0 R /XYZ 99.933071 124.788585 0 ] (cb17-22) [ 440 0 R /XYZ 99.933071 105.841563 0 ] (cb17-23) [ 464 0 R /XYZ 99.933071 799.173057 0 ] (cb17-24) [ 464 0 R /XYZ 99.933071 780.226036 0 ] (cb17-25) [ 464 0 R /XYZ 99.933071 761.279014 0 ] (cb17-26) [ 464 0 R /XYZ 99.933071 742.331993 0 ] (cb17-27) [ 464 0 R /XYZ 99.933071 723.384971 0 ] (cb17-28) [ 464 0 R /XYZ 99.933071 704.437950 0 ] (cb17-29) [ 464 0 R /XYZ 99.933071 685.490928 0 ] (cb17-3) [ 440 0 R /XYZ 99.933071 503.729014 0 ] (cb17-30) [ 464 0 R /XYZ 99.933071 666.543907 0 ] (cb17-31) [ 464 0 R /XYZ 99.933071 647.596885 0 ] (cb17-32) [ 464 0 R /XYZ 99.933071 628.649864 0 ] (cb17-33) [ 464 0 R /XYZ 99.933071 609.702842 0 ] (cb17-34) [ 464 0 R /XYZ 99.933071 590.755821 0 ] (cb17-35) [ 464 0 R /XYZ 99.933071 571.808799 0 ] (cb17-36) [ 464 0 R /XYZ 99.933071 552.861778 0 ] (cb17-37) [ 464 0 R /XYZ 99.933071 533.914756 0 ] (cb17-38) [ 464 0 R /XYZ 99.933071 514.967735 0 ] (cb17-39) [ 464 0 R /XYZ 99.933071 496.020714 0 ] (cb17-4) [ 440 0 R /XYZ 99.933071 465.834971 0 ] (cb17-40) [ 464 0 R /XYZ 99.933071 477.073692 0 ] (cb17-41) [ 464 0 R /XYZ 99.933071 458.126671 0 ] (cb17-42) [ 464 0 R /XYZ 99.933071 439.179649 0 ] (cb17-43) [ 464 0 R /XYZ 99.933071 420.232628 0 ] (cb17-44) [ 464 0 R /XYZ 99.933071 401.285606 0 ] (cb17-45) [ 464 0 R /XYZ 99.933071 382.338585 0 ] (cb17-46) [ 464 0 R /XYZ 99.933071 363.391563 0 ] (cb17-47) [ 464 0 R /XYZ 99.933071 344.444542 0 ] (cb17-48) [ 464 0 R /XYZ 99.933071 325.497520 0 ] (cb17-49) [ 464 0 R /XYZ 99.933071 306.550499 0 ] (cb17-5) [ 440 0 R /XYZ 99.933071 427.940928 0 ] (cb17-50) [ 464 0 R /XYZ 99.933071 287.603477 0 ] (cb17-51) [ 464 0 R /XYZ 99.933071 268.656456 0 ] (cb17-52) [ 464 0 R /XYZ 99.933071 249.709434 0 ] (cb17-53) [ 464 0 R /XYZ 99.933071 211.815391 0 ] (cb17-54) [ 464 0 R /XYZ 99.933071 192.868370 0 ] (cb17-55) [ 464 0 R /XYZ 99.933071 173.921348 0 ] (cb17-56) [ 464 0 R /XYZ 99.933071 154.974327 0 ] (cb17-57) [ 464 0 R /XYZ 99.933071 136.027305 0 ] (cb17-58) [ 464 0 R /XYZ 99.933071 117.080284 0 ] (cb17-59) [ 464 0 R /XYZ 99.933071 98.133262 0 ] (cb17-6) [ 440 0 R /XYZ 99.933071 408.993907 0 ] (cb17-60) [ 503 0 R /XYZ 99.933071 799.173057 0 ] (cb17-61) [ 503 0 R /XYZ 99.933071 780.226036 0 ] (cb17-7) [ 440 0 R /XYZ 99.933071 390.046885 0 ] (cb17-8) [ 440 0 R /XYZ 99.933071 371.099864 0 ] (cb17-9) [ 440 0 R /XYZ 99.933071 352.152842 0 ] (cb172) [ 751 0 R /XYZ 84.259843 607.820079 0 ] (cb172-1) [ 751 0 R /XYZ 101.433071 606.123057 0 ] (cb174) [ 751 0 R /XYZ 84.259843 298.679014 0 ] (cb174-1) [ 751 0 R /XYZ 101.433071 296.981993 0 ] (cb176) [ 755 0 R /XYZ 84.259843 715.370079 0 ] (cb176-1) [ 755 0 R /XYZ 101.433071 713.673057 0 ] (cb179) [ 755 0 R /XYZ 84.259843 375.479014 0 ] (cb179-1) [ 755 0 R /XYZ 101.433071 373.781993 0 ] (cb179-2) [ 755 0 R /XYZ 99.933071 354.834971 0 ] (cb179-3) [ 755 0 R /XYZ 99.933071 335.887950 0 ] (cb179-4) [ 755 0 R /XYZ 99.933071 279.046885 0 ] (cb179-5) [ 755 0 R /XYZ 99.933071 260.099864 0 ] (cb18) [ 503 0 R /XYZ 84.259843 720.976036 0 ] (cb18-1) [ 503 0 R /XYZ 101.433071 719.279014 0 ] (cb180) [ 755 0 R /XYZ 84.259843 200.849864 0 ] (cb180-1) [ 755 0 R /XYZ 101.433071 199.152842 0 ] (cb182) [ 764 0 R /XYZ 84.259843 655.220079 0 ] (cb182-1) [ 764 0 R /XYZ 101.433071 653.523057 0 ] (cb182-2) [ 764 0 R /XYZ 99.933071 634.576036 0 ] (cb182-3) [ 764 0 R /XYZ 99.933071 615.629014 0 ] (cb182-4) [ 764 0 R /XYZ 99.933071 558.787950 0 ] (cb182-5) [ 764 0 R /XYZ 99.933071 539.840928 0 ] (cb183) [ 764 0 R /XYZ 84.259843 480.590928 0 ] (cb183-1) [ 764 0 R /XYZ 101.433071 478.893907 0 ] (cb185) [ 774 0 R /XYZ 84.259843 490.251496 0 ] (cb185-1) [ 774 0 R /XYZ 101.433071 488.554475 0 ] (cb186) [ 774 0 R /XYZ 84.259843 429.304475 0 ] (cb186-1) [ 774 0 R /XYZ 101.433071 427.607453 0 ] (cb188) [ 774 0 R /XYZ 84.259843 116.807453 0 ] (cb188-1) [ 774 0 R /XYZ 101.433071 115.110432 0 ] (cb189) [ 779 0 R /XYZ 84.259843 772.370079 0 ] (cb189-1) [ 779 0 R /XYZ 101.433071 770.673057 0 ] (cb191) [ 779 0 R /XYZ 84.259843 459.873057 0 ] (cb191-1) [ 779 0 R /XYZ 101.433071 458.176036 0 ] (cb192) [ 779 0 R /XYZ 84.259843 398.926036 0 ] (cb192-1) [ 779 0 R /XYZ 101.433071 397.229014 0 ] (cb194) [ 784 0 R /XYZ 84.259843 799.370079 0 ] (cb194-1) [ 784 0 R /XYZ 101.433071 797.673057 0 ] (cb195) [ 784 0 R /XYZ 84.259843 738.423057 0 ] (cb195-1) [ 784 0 R /XYZ 101.433071 736.726036 0 ] (cb197) [ 784 0 R /XYZ 84.259843 410.926036 0 ] (cb197-1) [ 784 0 R /XYZ 101.433071 409.229014 0 ] (cb198) [ 784 0 R /XYZ 84.259843 349.979014 0 ] (cb198-1) [ 784 0 R /XYZ 101.433071 348.281993 0 ] (cb2) [ 318 0 R /XYZ 84.259843 730.370079 0 ] (cb2-1) [ 318 0 R /XYZ 101.433071 728.673057 0 ] (cb201) [ 790 0 R /XYZ 84.259843 230.570079 0 ] (cb201-1) [ 790 0 R /XYZ 101.433071 228.873057 0 ] (cb202) [ 790 0 R /XYZ 84.259843 169.623057 0 ] (cb202-1) [ 790 0 R /XYZ 101.433071 167.926036 0 ] (cb204) [ 794 0 R /XYZ 84.259843 583.970079 0 ] (cb204-1) [ 794 0 R /XYZ 101.433071 582.273057 0 ] (cb205) [ 794 0 R /XYZ 84.259843 523.023057 0 ] (cb205-1) [ 794 0 R /XYZ 101.433071 521.326036 0 ] (cb207) [ 798 0 R /XYZ 84.259843 700.370079 0 ] (cb207-1) [ 798 0 R /XYZ 101.433071 698.673057 0 ] (cb208) [ 798 0 R /XYZ 84.259843 639.423057 0 ] (cb208-1) [ 798 0 R /XYZ 101.433071 637.726036 0 ] (cb21) [ 508 0 R /XYZ 84.259843 565.251496 0 ] (cb21-1) [ 508 0 R /XYZ 101.433071 563.554475 0 ] (cb210) [ 798 0 R /XYZ 84.259843 296.926036 0 ] (cb210-1) [ 798 0 R /XYZ 101.433071 295.229014 0 ] (cb211) [ 798 0 R /XYZ 84.259843 235.979014 0 ] (cb211-1) [ 798 0 R /XYZ 101.433071 234.281993 0 ] (cb213) [ 804 0 R /XYZ 84.259843 640.370079 0 ] (cb213-1) [ 804 0 R /XYZ 101.433071 638.673057 0 ] (cb214) [ 804 0 R /XYZ 84.259843 579.423057 0 ] (cb214-1) [ 804 0 R /XYZ 101.433071 577.726036 0 ] (cb217) [ 808 0 R /XYZ 84.259843 428.570079 0 ] (cb217-1) [ 808 0 R /XYZ 101.433071 426.873057 0 ] (cb218) [ 808 0 R /XYZ 84.259843 131.823057 0 ] (cb218-1) [ 808 0 R /XYZ 101.433071 130.126036 0 ] (cb219) [ 812 0 R /XYZ 84.259843 799.370079 0 ] (cb219-1) [ 812 0 R /XYZ 101.433071 797.673057 0 ] (cb221) [ 817 0 R /XYZ 84.259843 768.620079 0 ] (cb221-1) [ 817 0 R /XYZ 101.433071 766.923057 0 ] (cb221-10) [ 817 0 R /XYZ 99.933071 596.399864 0 ] (cb221-11) [ 817 0 R /XYZ 99.933071 577.452842 0 ] (cb221-12) [ 817 0 R /XYZ 99.933071 482.717735 0 ] (cb221-13) [ 817 0 R /XYZ 99.933071 463.770714 0 ] (cb221-14) [ 817 0 R /XYZ 99.933071 444.823692 0 ] (cb221-15) [ 817 0 R /XYZ 99.933071 425.876671 0 ] (cb221-16) [ 817 0 R /XYZ 99.933071 406.929649 0 ] (cb221-17) [ 817 0 R /XYZ 99.933071 387.982628 0 ] (cb221-18) [ 817 0 R /XYZ 99.933071 350.088585 0 ] (cb221-19) [ 817 0 R /XYZ 99.933071 331.141563 0 ] (cb221-2) [ 817 0 R /XYZ 99.933071 747.976036 0 ] (cb221-20) [ 817 0 R /XYZ 99.933071 198.512413 0 ] (cb221-21) [ 817 0 R /XYZ 99.933071 179.565391 0 ] (cb221-22) [ 817 0 R /XYZ 99.933071 160.618370 0 ] (cb221-23) [ 817 0 R /XYZ 99.933071 103.777305 0 ] (cb221-24) [ 842 0 R /XYZ 99.933071 799.173057 0 ] (cb221-25) [ 842 0 R /XYZ 99.933071 780.226036 0 ] (cb221-26) [ 842 0 R /XYZ 99.933071 761.279014 0 ] (cb221-27) [ 842 0 R /XYZ 99.933071 742.331993 0 ] (cb221-28) [ 842 0 R /XYZ 99.933071 723.384971 0 ] (cb221-29) [ 842 0 R /XYZ 99.933071 704.437950 0 ] (cb221-3) [ 817 0 R /XYZ 99.933071 729.029014 0 ] (cb221-30) [ 842 0 R /XYZ 99.933071 685.490928 0 ] (cb221-31) [ 842 0 R /XYZ 99.933071 666.543907 0 ] (cb221-4) [ 817 0 R /XYZ 99.933071 710.081993 0 ] (cb221-5) [ 817 0 R /XYZ 99.933071 691.134971 0 ] (cb221-6) [ 817 0 R /XYZ 99.933071 672.187950 0 ] (cb221-7) [ 817 0 R /XYZ 99.933071 653.240928 0 ] (cb221-8) [ 817 0 R /XYZ 99.933071 634.293907 0 ] (cb221-9) [ 817 0 R /XYZ 99.933071 615.346885 0 ] (cb222) [ 842 0 R /XYZ 84.259843 607.293907 0 ] (cb222-1) [ 842 0 R /XYZ 101.433071 605.596885 0 ] (cb225) [ 853 0 R /XYZ 84.259843 434.451496 0 ] (cb225-1) [ 853 0 R /XYZ 101.433071 432.754475 0 ] (cb228) [ 856 0 R /XYZ 84.259843 799.370079 0 ] (cb228-1) [ 856 0 R /XYZ 101.433071 797.673057 0 ] (cb231) [ 856 0 R /XYZ 84.259843 448.229014 0 ] (cb231-1) [ 856 0 R /XYZ 101.433071 446.531993 0 ] (cb234) [ 860 0 R /XYZ 84.259843 799.370079 0 ] (cb234-1) [ 860 0 R /XYZ 101.433071 797.673057 0 ] (cb237) [ 863 0 R /XYZ 84.259843 394.701496 0 ] (cb237-1) [ 863 0 R /XYZ 101.433071 393.004475 0 ] (cb237-10) [ 863 0 R /XYZ 99.933071 222.481281 0 ] (cb237-11) [ 863 0 R /XYZ 99.933071 203.534260 0 ] (cb237-12) [ 863 0 R /XYZ 99.933071 165.640217 0 ] (cb237-13) [ 863 0 R /XYZ 99.933071 127.746174 0 ] (cb237-14) [ 863 0 R /XYZ 99.933071 108.799152 0 ] (cb237-15) [ 863 0 R /XYZ 99.933071 89.852131 0 ] (cb237-16) [ 880 0 R /XYZ 99.933071 799.173057 0 ] (cb237-17) [ 880 0 R /XYZ 99.933071 780.226036 0 ] (cb237-18) [ 880 0 R /XYZ 99.933071 761.279014 0 ] (cb237-19) [ 880 0 R /XYZ 99.933071 742.331993 0 ] (cb237-2) [ 863 0 R /XYZ 99.933071 374.057453 0 ] (cb237-3) [ 863 0 R /XYZ 99.933071 355.110432 0 ] (cb237-4) [ 863 0 R /XYZ 99.933071 336.163410 0 ] (cb237-5) [ 863 0 R /XYZ 99.933071 317.216389 0 ] (cb237-6) [ 863 0 R /XYZ 99.933071 298.269367 0 ] (cb237-7) [ 863 0 R /XYZ 99.933071 279.322346 0 ] (cb237-8) [ 863 0 R /XYZ 99.933071 260.375324 0 ] (cb237-9) [ 863 0 R /XYZ 99.933071 241.428303 0 ] (cb238) [ 880 0 R /XYZ 84.259843 683.081993 0 ] (cb238-1) [ 880 0 R /XYZ 101.433071 681.384971 0 ] (cb24) [ 511 0 R /XYZ 84.259843 125.570079 0 ] (cb24-1) [ 511 0 R /XYZ 101.433071 123.873057 0 ] (cb241) [ 893 0 R /XYZ 84.259843 768.620079 0 ] (cb241-1) [ 893 0 R /XYZ 101.433071 766.923057 0 ] (cb241-10) [ 893 0 R /XYZ 99.933071 596.399864 0 ] (cb241-11) [ 893 0 R /XYZ 99.933071 577.452842 0 ] (cb241-12) [ 893 0 R /XYZ 99.933071 520.611778 0 ] (cb241-13) [ 893 0 R /XYZ 99.933071 501.664756 0 ] (cb241-14) [ 893 0 R /XYZ 99.933071 482.717735 0 ] (cb241-15) [ 893 0 R /XYZ 99.933071 463.770714 0 ] (cb241-16) [ 893 0 R /XYZ 99.933071 444.823692 0 ] (cb241-17) [ 893 0 R /XYZ 99.933071 425.876671 0 ] (cb241-18) [ 893 0 R /XYZ 99.933071 406.929649 0 ] (cb241-2) [ 893 0 R /XYZ 99.933071 747.976036 0 ] (cb241-3) [ 893 0 R /XYZ 99.933071 729.029014 0 ] (cb241-4) [ 893 0 R /XYZ 99.933071 710.081993 0 ] (cb241-5) [ 893 0 R /XYZ 99.933071 691.134971 0 ] (cb241-6) [ 893 0 R /XYZ 99.933071 672.187950 0 ] (cb241-7) [ 893 0 R /XYZ 99.933071 653.240928 0 ] (cb241-8) [ 893 0 R /XYZ 99.933071 634.293907 0 ] (cb241-9) [ 893 0 R /XYZ 99.933071 615.346885 0 ] (cb242) [ 893 0 R /XYZ 84.259843 347.679649 0 ] (cb242-1) [ 893 0 R /XYZ 101.433071 345.982628 0 ] (cb27) [ 516 0 R /XYZ 84.259843 425.570079 0 ] (cb27-1) [ 516 0 R /XYZ 101.433071 423.873057 0 ] (cb3) [ 318 0 R /XYZ 84.259843 523.623057 0 ] (cb3-1) [ 318 0 R /XYZ 101.433071 521.926036 0 ] (cb30) [ 521 0 R /XYZ 84.259843 715.370079 0 ] (cb30-1) [ 521 0 R /XYZ 101.433071 713.673057 0 ] (cb33) [ 524 0 R /XYZ 84.259843 211.820079 0 ] (cb33-1) [ 524 0 R /XYZ 101.433071 210.123057 0 ] (cb36) [ 529 0 R /XYZ 84.259843 451.820079 0 ] (cb36-1) [ 529 0 R /XYZ 101.433071 450.123057 0 ] (cb39) [ 534 0 R /XYZ 84.259843 730.370079 0 ] (cb39-1) [ 534 0 R /XYZ 101.433071 728.673057 0 ] (cb42) [ 537 0 R /XYZ 84.259843 294.320079 0 ] (cb42-1) [ 537 0 R /XYZ 101.433071 292.623057 0 ] (cb45) [ 542 0 R /XYZ 84.259843 575.570079 0 ] (cb45-1) [ 542 0 R /XYZ 101.433071 573.873057 0 ] (cb48) [ 545 0 R /XYZ 84.259843 129.320079 0 ] (cb48-1) [ 545 0 R /XYZ 101.433071 127.623057 0 ] (cb51) [ 550 0 R /XYZ 84.259843 350.570079 0 ] (cb51-1) [ 550 0 R /XYZ 101.433071 348.873057 0 ] (cb54) [ 555 0 R /XYZ 84.259843 538.070079 0 ] (cb54-1) [ 555 0 R /XYZ 101.433071 536.373057 0 ] (cb55) [ 555 0 R /XYZ 84.259843 301.323057 0 ] (cb55-1) [ 555 0 R /XYZ 101.433071 299.626036 0 ] (cb56) [ 559 0 R /XYZ 84.259843 799.370079 0 ] (cb56-1) [ 559 0 R /XYZ 101.433071 797.673057 0 ] (cb59) [ 562 0 R /XYZ 84.259843 350.570079 0 ] (cb59-1) [ 562 0 R /XYZ 101.433071 348.873057 0 ] (cb6) [ 318 0 R /XYZ 84.259843 138.376036 0 ] (cb6-1) [ 318 0 R /XYZ 101.433071 136.679014 0 ] (cb62) [ 567 0 R /XYZ 84.259843 598.970079 0 ] (cb62-1) [ 567 0 R /XYZ 101.433071 597.273057 0 ] (cb65) [ 572 0 R /XYZ 84.259843 772.370079 0 ] (cb65-1) [ 572 0 R /XYZ 101.433071 770.673057 0 ] (cb68) [ 575 0 R /XYZ 84.259843 271.820079 0 ] (cb68-1) [ 575 0 R /XYZ 101.433071 270.123057 0 ] (cb71) [ 580 0 R /XYZ 84.259843 571.820079 0 ] (cb71-1) [ 580 0 R /XYZ 101.433071 570.123057 0 ] (cb74) [ 583 0 R /XYZ 84.259843 144.320079 0 ] (cb74-1) [ 583 0 R /XYZ 101.433071 142.623057 0 ] (cb77) [ 588 0 R /XYZ 84.259843 440.570079 0 ] (cb77-1) [ 588 0 R /XYZ 101.433071 438.873057 0 ] (cb80) [ 593 0 R /XYZ 84.259843 715.370079 0 ] (cb80-1) [ 593 0 R /XYZ 101.433071 713.673057 0 ] (cb83) [ 596 0 R /XYZ 84.259843 275.570079 0 ] (cb83-1) [ 596 0 R /XYZ 101.433071 273.873057 0 ] (cb86) [ 601 0 R /XYZ 84.259843 556.820079 0 ] (cb86-1) [ 601 0 R /XYZ 101.433071 555.123057 0 ] (cb89) [ 604 0 R /XYZ 84.259843 125.570079 0 ] (cb89-1) [ 604 0 R /XYZ 101.433071 123.873057 0 ] (cb9) [ 325 0 R /XYZ 84.259843 509.420079 0 ] (cb9-1) [ 325 0 R /XYZ 101.433071 507.723057 0 ] (cb9-10) [ 325 0 R /XYZ 99.933071 318.252842 0 ] (cb9-11) [ 325 0 R /XYZ 99.933071 299.305821 0 ] (cb9-12) [ 325 0 R /XYZ 99.933071 280.358799 0 ] (cb9-13) [ 325 0 R /XYZ 99.933071 261.411778 0 ] (cb9-14) [ 325 0 R /XYZ 99.933071 242.464756 0 ] (cb9-15) [ 325 0 R /XYZ 99.933071 223.517735 0 ] (cb9-16) [ 325 0 R /XYZ 99.933071 204.570714 0 ] (cb9-17) [ 325 0 R /XYZ 99.933071 185.623692 0 ] (cb9-18) [ 325 0 R /XYZ 99.933071 166.676671 0 ] (cb9-19) [ 325 0 R /XYZ 99.933071 147.729649 0 ] (cb9-2) [ 325 0 R /XYZ 99.933071 488.776036 0 ] (cb9-20) [ 325 0 R /XYZ 99.933071 128.782628 0 ] (cb9-21) [ 325 0 R /XYZ 99.933071 109.835606 0 ] (cb9-22) [ 325 0 R /XYZ 99.933071 90.888585 0 ] (cb9-23) [ 349 0 R /XYZ 99.933071 799.173057 0 ] (cb9-24) [ 349 0 R /XYZ 99.933071 780.226036 0 ] (cb9-25) [ 349 0 R /XYZ 99.933071 761.279014 0 ] (cb9-26) [ 349 0 R /XYZ 99.933071 742.331993 0 ] (cb9-27) [ 349 0 R /XYZ 99.933071 723.384971 0 ] (cb9-28) [ 349 0 R /XYZ 99.933071 704.437950 0 ] (cb9-29) [ 349 0 R /XYZ 99.933071 685.490928 0 ] (cb9-3) [ 325 0 R /XYZ 99.933071 469.829014 0 ] (cb9-30) [ 349 0 R /XYZ 99.933071 666.543907 0 ] (cb9-31) [ 349 0 R /XYZ 99.933071 647.596885 0 ] (cb9-32) [ 349 0 R /XYZ 99.933071 628.649864 0 ] (cb9-33) [ 349 0 R /XYZ 99.933071 609.702842 0 ] (cb9-34) [ 349 0 R /XYZ 99.933071 590.755821 0 ] (cb9-35) [ 349 0 R /XYZ 99.933071 571.808799 0 ] (cb9-36) [ 349 0 R /XYZ 99.933071 552.861778 0 ] (cb9-37) [ 349 0 R /XYZ 99.933071 533.914756 0 ] (cb9-38) [ 349 0 R /XYZ 99.933071 514.967735 0 ] (cb9-39) [ 349 0 R /XYZ 99.933071 496.020714 0 ] (cb9-4) [ 325 0 R /XYZ 99.933071 431.934971 0 ] (cb9-40) [ 349 0 R /XYZ 99.933071 477.073692 0 ] (cb9-41) [ 349 0 R /XYZ 99.933071 458.126671 0 ] (cb9-42) [ 349 0 R /XYZ 99.933071 439.179649 0 ] (cb9-43) [ 349 0 R /XYZ 99.933071 420.232628 0 ] (cb9-44) [ 349 0 R /XYZ 99.933071 401.285606 0 ] (cb9-5) [ 325 0 R /XYZ 99.933071 412.987950 0 ] (cb9-6) [ 325 0 R /XYZ 99.933071 394.040928 0 ] (cb9-7) [ 325 0 R /XYZ 99.933071 375.093907 0 ] (cb9-8) [ 325 0 R /XYZ 99.933071 356.146885 0 ] (cb9-9) [ 325 0 R /XYZ 99.933071 337.199864 0 ] (cb92) [ 609 0 R /XYZ 84.259843 440.570079 0 ] (cb92-1) [ 609 0 R /XYZ 101.433071 438.873057 0 ] (cb95) [ 614 0 R /XYZ 84.259843 685.370079 0 ] (cb95-1) [ 614 0 R /XYZ 101.433071 683.673057 0 ] (cb98) [ 617 0 R /XYZ 84.259843 238.070079 0 ] (cb98-1) [ 617 0 R /XYZ 101.433071 236.373057 0 ] (cis-v1.6-kubernetes-benchmark---rancher-v2.6-with-kubernetes-v1.18-to-v1.21) [ 304 0 R /XYZ 84.259843 784.970079 0 ] (client-certificate-authentication-should-not-be-used-for-users-manual) [ 741 0 R /XYZ 84.259843 684.651496 0 ] (configure-image-provenance-using-imagepolicywebhook-admission-controller-manual) [ 889 0 R /XYZ 84.259843 684.651496 0 ] (consider-external-secret-storage-manual) [ 887 0 R /XYZ 84.259843 568.851496 0 ] (contentsbox) [ 8 0 R /XYZ 77.509843 756.850394 0 ] (controller-manager) [ 674 0 R /XYZ 41.740157 784.430079 0 ] (controls) [ 309 0 R /XYZ 84.259843 612.381496 0 ] (create-administrative-boundaries-between-resources-using-namespaces-manual) [ 891 0 R /XYZ 84.259843 720.651496 0 ] (ensure-that-a-minimal-audit-policy-is-created-automated) [ 743 0 R /XYZ 84.259843 720.651496 0 ] (ensure-that-a-unique-certificate-authority-is-used-for-etcd-automated) [ 734 0 R /XYZ 84.259843 799.370079 0 ] (ensure-that-all-namespaces-have-network-policies-defined-automated) [ 863 0 R /XYZ 84.259843 538.851496 0 ] (ensure-that-default-service-accounts-are-not-actively-used.-automated) [ 815 0 R /XYZ 84.259843 206.451496 0 ] (ensure-that-encryption-providers-are-appropriately-configured-automated) [ 643 0 R /XYZ 84.259843 577.220079 0 ] (ensure-that-service-account-tokens-are-only-mounted-where-necessary-manual) [ 842 0 R /XYZ 84.259843 449.446885 0 ] (ensure-that-the---anonymous-auth-argument-is-set-to-false-automated) [ 508 0 R /XYZ 84.259843 720.651496 0 ] (ensure-that-the---audit-log-maxage-argument-is-set-to-30-or-as-appropriate-automated) [ 596 0 R /XYZ 84.259843 445.970079 0 ] (ensure-that-the---audit-log-maxbackup-argument-is-set-to-10-or-as-appropriate-automated) [ 601 0 R /XYZ 84.259843 727.220079 0 ] (ensure-that-the---audit-log-maxsize-argument-is-set-to-100-or-as-appropriate-automated) [ 604 0 R /XYZ 84.259843 295.970079 0 ] (ensure-that-the---audit-log-path-argument-is-set-automated) [ 591 0 R /XYZ 84.259843 183.470079 0 ] (ensure-that-the---authorization-mode-argument-includes-node-automated) [ 537 0 R /XYZ 84.259843 464.720079 0 ] (ensure-that-the---authorization-mode-argument-includes-rbac-automated) [ 542 0 R /XYZ 84.259843 745.970079 0 ] (ensure-that-the---authorization-mode-argument-is-not-set-to-alwaysallow-automated) [ 532 0 R /XYZ 84.259843 183.470079 0 ] (ensure-that-the---authorization-mode-argument-is-not-set-to-alwaysallow-automated-1) [ 774 0 R /XYZ 84.259843 332.207453 0 ] (ensure-that-the---auto-tls-argument-is-not-set-to-true-automated) [ 710 0 R /XYZ 84.259843 258.470079 0 ] (ensure-that-the---basic-auth-file-argument-is-not-set-automated) [ 511 0 R /XYZ 84.259843 295.970079 0 ] (ensure-that-the---bind-address-argument-is-set-to-127.0.0.1-automated) [ 689 0 R /XYZ 84.259843 524.420079 0 ] (ensure-that-the---bind-address-argument-is-set-to-127.0.0.1-automated-1) [ 694 0 R /XYZ 84.259843 257.404475 0 ] (ensure-that-the---cert-file-and---key-file-arguments-are-set-as-appropriate-automated) [ 700 0 R /XYZ 84.259843 684.651496 0 ] (ensure-that-the---client-ca-file-argument-is-set-as-appropriate-automated) [ 630 0 R /XYZ 84.259843 483.470079 0 ] (ensure-that-the---client-ca-file-argument-is-set-as-appropriate-automated-1) [ 779 0 R /XYZ 84.259843 675.273057 0 ] (ensure-that-the---client-cert-auth-argument-is-set-to-true-automated) [ 705 0 R /XYZ 84.259843 464.720079 0 ] (ensure-that-the---encryption-provider-config-argument-is-set-as-appropriate-automated) [ 638 0 R /XYZ 84.259843 295.970079 0 ] (ensure-that-the---etcd-cafile-argument-is-set-as-appropriate-automated) [ 635 0 R /XYZ 84.259843 745.970079 0 ] (ensure-that-the---etcd-certfile-and---etcd-keyfile-arguments-are-set-as-appropriate-automated) [ 622 0 R /XYZ 84.259843 689.720079 0 ] (ensure-that-the---event-qps-argument-is-set-to-0-or-a-level-which-ensures-appropriate-event-capture-automated) [ 794 0 R /XYZ 84.259843 208.126036 0 ] (ensure-that-the---hostname-override-argument-is-not-set-manual) [ 794 0 R /XYZ 84.259843 425.926036 0 ] (ensure-that-the---insecure-bind-address-argument-is-not-set-automated) [ 575 0 R /XYZ 84.259843 427.220079 0 ] (ensure-that-the---insecure-port-argument-is-set-to-0-automated) [ 580 0 R /XYZ 84.259843 727.220079 0 ] (ensure-that-the---kubeconfig-kubelet.conf-file-ownership-is-set-to-rootroot-automated) [ 751 0 R /XYZ 84.259843 163.687950 0 ] (ensure-that-the---kubeconfig-kubelet.conf-file-permissions-are-set-to-644-or-more-restrictive-automated) [ 751 0 R /XYZ 84.259843 454.079014 0 ] (ensure-that-the---kubelet-certificate-authority-argument-is-set-as-appropriate-automated) [ 529 0 R /XYZ 84.259843 652.220079 0 ] (ensure-that-the---kubelet-client-certificate-and---kubelet-client-key-arguments-are-set-as-appropriate-automated) [ 524 0 R /XYZ 84.259843 427.220079 0 ] (ensure-that-the---kubelet-https-argument-is-set-to-true-automated) [ 519 0 R /XYZ 84.259843 164.720079 0 ] (ensure-that-the---make-iptables-util-chains-argument-is-set-to-true-automated) [ 794 0 R /XYZ 84.259843 799.370079 0 ] (ensure-that-the---peer-auto-tls-argument-is-not-set-to-true-automated) [ 727 0 R /XYZ 84.259843 333.470079 0 ] (ensure-that-the---peer-cert-file-and---peer-key-file-arguments-are-set-as-appropriate-automated) [ 717 0 R /XYZ 84.259843 799.370079 0 ] (ensure-that-the---peer-client-cert-auth-argument-is-set-to-true-automated) [ 722 0 R /XYZ 84.259843 558.470079 0 ] (ensure-that-the---profiling-argument-is-set-to-false-automated) [ 588 0 R /XYZ 84.259843 595.970079 0 ] (ensure-that-the---profiling-argument-is-set-to-false-automated-1) [ 677 0 R /XYZ 84.259843 727.220079 0 ] (ensure-that-the---profiling-argument-is-set-to-false-automated-2) [ 694 0 R /XYZ 84.259843 720.651496 0 ] (ensure-that-the---protect-kernel-defaults-argument-is-set-to-true-automated) [ 790 0 R /XYZ 84.259843 445.970079 0 ] (ensure-that-the---read-only-port-argument-is-set-to-0-automated) [ 779 0 R /XYZ 84.259843 301.829014 0 ] (ensure-that-the---request-timeout-argument-is-set-as-appropriate-automated) [ 609 0 R /XYZ 84.259843 595.970079 0 ] (ensure-that-the---root-ca-file-argument-is-set-as-appropriate-automated) [ 686 0 R /XYZ 84.259843 727.220079 0 ] (ensure-that-the---rotate-certificates-argument-is-not-set-to-false-automated) [ 798 0 R /XYZ 84.259843 138.881993 0 ] (ensure-that-the---secure-port-argument-is-not-set-to-0-automated) [ 583 0 R /XYZ 84.259843 314.720079 0 ] (ensure-that-the---service-account-key-file-argument-is-set-as-appropriate-automated) [ 617 0 R /XYZ 84.259843 408.470079 0 ] (ensure-that-the---service-account-lookup-argument-is-set-to-true-automated) [ 612 0 R /XYZ 84.259843 164.720079 0 ] (ensure-that-the---service-account-private-key-file-argument-is-set-as-appropriate-automated) [ 683 0 R /XYZ 84.259843 745.970079 0 ] (ensure-that-the---streaming-connection-idle-timeout-argument-is-not-set-to-0-automated) [ 784 0 R /XYZ 84.259843 641.326036 0 ] (ensure-that-the---terminated-pod-gc-threshold-argument-is-set-as-appropriate-automated) [ 674 0 R /XYZ 84.259843 720.651496 0 ] (ensure-that-the---tls-cert-file-and---tls-private-key-file-arguments-are-set-as-appropriate-automated) [ 625 0 R /XYZ 84.259843 239.720079 0 ] (ensure-that-the---tls-cert-file-and---tls-private-key-file-arguments-are-set-as-appropriate-automated-1) [ 798 0 R /XYZ 84.259843 542.326036 0 ] (ensure-that-the---token-auth-file-parameter-is-not-set-automated) [ 516 0 R /XYZ 84.259843 595.970079 0 ] (ensure-that-the---use-service-account-credentials-argument-is-set-to-true-automated) [ 680 0 R /XYZ 84.259843 745.970079 0 ] (ensure-that-the-admin.conf-file-ownership-is-set-to-rootroot-automated) [ 323 0 R /XYZ 84.259843 586.670079 0 ] (ensure-that-the-admin.conf-file-permissions-are-set-to-644-or-more-restrictive-automated) [ 323 0 R /XYZ 84.259843 702.470079 0 ] (ensure-that-the-admission-control-plugin-alwaysadmit-is-not-set-automated) [ 550 0 R /XYZ 84.259843 520.970079 0 ] (ensure-that-the-admission-control-plugin-alwayspullimages-is-set-manual) [ 555 0 R /XYZ 84.259843 708.470079 0 ] (ensure-that-the-admission-control-plugin-eventratelimit-is-set-automated) [ 545 0 R /XYZ 84.259843 314.720079 0 ] (ensure-that-the-admission-control-plugin-namespacelifecycle-is-set-automated) [ 562 0 R /XYZ 84.259843 520.970079 0 ] (ensure-that-the-admission-control-plugin-noderestriction-is-set-automated) [ 570 0 R /XYZ 84.259843 258.470079 0 ] (ensure-that-the-admission-control-plugin-podsecuritypolicy-is-set-automated) [ 567 0 R /XYZ 84.259843 799.370079 0 ] (ensure-that-the-admission-control-plugin-securitycontextdeny-is-set-if-podsecuritypolicy-is-not-used-manual) [ 555 0 R /XYZ 84.259843 501.723057 0 ] (ensure-that-the-admission-control-plugin-serviceaccount-is-set-automated) [ 555 0 R /XYZ 84.259843 264.976036 0 ] (ensure-that-the-anonymous-auth-argument-is-set-to-false-automated) [ 774 0 R /XYZ 84.259843 720.651496 0 ] (ensure-that-the-api-server-only-makes-use-of-strong-cryptographic-ciphers-automated) [ 660 0 R /XYZ 84.259843 411.552842 0 ] (ensure-that-the-api-server-pod-specification-file-ownership-is-set-to-rootroot-automated) [ 313 0 R /XYZ 84.259843 553.851496 0 ] (ensure-that-the-api-server-pod-specification-file-permissions-are-set-to-644-or-more-restrictive-automated) [ 313 0 R /XYZ 84.259843 684.651496 0 ] (ensure-that-the-audit-policy-covers-key-security-concerns-manual) [ 746 0 R /XYZ 84.259843 333.470079 0 ] (ensure-that-the-certificate-authorities-file-permissions-are-set-to-644-or-more-restrictive-automated) [ 755 0 R /XYZ 84.259843 519.629014 0 ] (ensure-that-the-client-certificate-authorities-file-ownership-is-set-to-rootroot-automated) [ 764 0 R /XYZ 84.259843 799.370079 0 ] (ensure-that-the-cluster-admin-role-is-only-used-where-required-manual) [ 815 0 R /XYZ 84.259843 684.651496 0 ] (ensure-that-the-cni-in-use-supports-network-policies-manual) [ 863 0 R /XYZ 84.259843 684.651496 0 ] (ensure-that-the-container-network-interface-file-ownership-is-set-to-rootroot-manual) [ 315 0 R /XYZ 84.259843 170.823057 0 ] (ensure-that-the-container-network-interface-file-permissions-are-set-to-644-or-more-restrictive-manual) [ 315 0 R /XYZ 84.259843 347.570079 0 ] (ensure-that-the-controller-manager-pod-specification-file-ownership-is-set-to-rootroot-automated) [ 313 0 R /XYZ 84.259843 277.251496 0 ] (ensure-that-the-controller-manager-pod-specification-file-permissions-are-set-to-644-or-more-restrictive-automated) [ 313 0 R /XYZ 84.259843 423.051496 0 ] (ensure-that-the-controller-manager.conf-file-ownership-is-set-to-rootroot-automated) [ 325 0 R /XYZ 84.259843 799.370079 0 ] (ensure-that-the-controller-manager.conf-file-permissions-are-set-to-644-or-more-restrictive-automated) [ 323 0 R /XYZ 84.259843 209.270079 0 ] (ensure-that-the-etcd-data-directory-ownership-is-set-to-etcdetcd-automated) [ 318 0 R /XYZ 84.259843 365.776036 0 ] (ensure-that-the-etcd-data-directory-permissions-are-set-to-700-or-more-restrictive-automated) [ 318 0 R /XYZ 84.259843 694.023057 0 ] (ensure-that-the-etcd-pod-specification-file-ownership-is-set-to-rootroot-automated) [ 315 0 R /XYZ 84.259843 478.370079 0 ] (ensure-that-the-etcd-pod-specification-file-permissions-are-set-to-644-or-more-restrictive-automated) [ 315 0 R /XYZ 84.259843 609.170079 0 ] (ensure-that-the-kubelet---config-configuration-file-has-permissions-set-to-644-or-more-restrictive-automated) [ 764 0 R /XYZ 84.259843 383.493907 0 ] (ensure-that-the-kubelet---config-configuration-file-ownership-is-set-to-rootroot-automated) [ 764 0 R /XYZ 84.259843 195.693907 0 ] (ensure-that-the-kubelet-only-makes-use-of-strong-cryptographic-ciphers-automated) [ 808 0 R /XYZ 84.259843 392.223057 0 ] (ensure-that-the-kubelet-service-file-ownership-is-set-to-rootroot-automated) [ 748 0 R /XYZ 84.259843 553.851496 0 ] (ensure-that-the-kubelet-service-file-permissions-are-set-to-644-or-more-restrictive-automated) [ 748 0 R /XYZ 84.259843 684.651496 0 ] (ensure-that-the-kubernetes-pki-certificate-file-permissions-are-set-to-644-or-more-restrictive-automated) [ 349 0 R /XYZ 84.259843 184.188585 0 ] (ensure-that-the-kubernetes-pki-directory-and-file-ownership-is-set-to-rootroot-automated) [ 325 0 R /XYZ 84.259843 668.570079 0 ] (ensure-that-the-kubernetes-pki-key-file-permissions-are-set-to-600-automated) [ 440 0 R /XYZ 84.259843 702.470079 0 ] (ensure-that-the-proxy-kubeconfig-file-ownership-is-set-to-rootroot-automated) [ 751 0 R /XYZ 84.259843 763.220079 0 ] (ensure-that-the-rotatekubeletservercertificate-argument-is-set-to-true-automated) [ 689 0 R /XYZ 84.259843 727.220079 0 ] (ensure-that-the-scheduler-pod-specification-file-ownership-is-set-to-rootroot-automated) [ 315 0 R /XYZ 84.259843 739.970079 0 ] (ensure-that-the-scheduler-pod-specification-file-permissions-are-set-to-644-or-more-restrictive-automated) [ 313 0 R /XYZ 84.259843 146.451496 0 ] (ensure-that-the-scheduler.conf-file-ownership-is-set-to-rootroot-automated) [ 323 0 R /XYZ 84.259843 340.070079 0 ] (ensure-that-the-scheduler.conf-file-permissions-are-set-to-644-or-more-restrictive-automated) [ 323 0 R /XYZ 84.259843 470.870079 0 ] (ensure-that-the-seccomp-profile-is-set-to-dockerdefault-in-your-pod-definitions-manual) [ 891 0 R /XYZ 84.259843 604.851496 0 ] (etcd-node-configuration-files) [ 700 0 R /XYZ 41.740157 784.430079 0 ] (extensible-admission-control) [ 889 0 R /XYZ 41.740157 784.430079 0 ] (general-policies) [ 891 0 R /XYZ 41.740157 784.430079 0 ] (header_bottom_text) [ 6 0 R /XYZ 85.009843 392.718898 0 ] (if-proxy-kubeconfig-file-exists-ensure-permissions-are-set-to-644-or-more-restrictive-automated) [ 748 0 R /XYZ 84.259843 423.051496 0 ] (kubelet) [ 774 0 R /XYZ 41.740157 784.430079 0 ] (logging) [ 743 0 R /XYZ 41.740157 784.430079 0 ] (master-node-configuration-files) [ 313 0 R /XYZ 41.740157 784.430079 0 ] (minimize-access-to-create-pods-manual) [ 815 0 R /XYZ 84.259843 307.251496 0 ] (minimize-access-to-secrets-manual) [ 815 0 R /XYZ 84.259843 523.851496 0 ] (minimize-the-admission-of-containers-wishing-to-share-the-host-ipc-namespace-automated) [ 853 0 R /XYZ 84.259843 238.710432 0 ] (minimize-the-admission-of-containers-wishing-to-share-the-host-network-namespace-automated) [ 856 0 R /XYZ 84.259843 603.629014 0 ] (minimize-the-admission-of-containers-wishing-to-share-the-host-process-id-namespace-automated) [ 853 0 R /XYZ 84.259843 589.851496 0 ] (minimize-the-admission-of-containers-with-added-capabilities-manual) [ 860 0 R /XYZ 84.259843 323.081993 0 ] (minimize-the-admission-of-containers-with-allowprivilegeescalation-automated) [ 856 0 R /XYZ 84.259843 252.487950 0 ] (minimize-the-admission-of-containers-with-capabilities-assigned-manual) [ 860 0 R /XYZ 84.259843 207.281993 0 ] (minimize-the-admission-of-containers-with-the-net_raw-capability-manual) [ 860 0 R /XYZ 84.259843 453.881993 0 ] (minimize-the-admission-of-privileged-containers-manual) [ 853 0 R /XYZ 84.259843 720.651496 0 ] (minimize-the-admission-of-root-containers-manual) [ 860 0 R /XYZ 84.259843 584.681993 0 ] (minimize-wildcard-use-in-roles-and-clusterroles-manual) [ 815 0 R /XYZ 84.259843 423.051496 0 ] (network-policies-and-cni) [ 863 0 R /XYZ 41.740157 784.430079 0 ] (overview) [ 304 0 R /XYZ 84.259843 704.822079 0 ] (pod-security-policies) [ 853 0 R /XYZ 41.740157 784.430079 0 ] (prefer-using-secrets-as-files-over-secrets-as-environment-variables-manual) [ 887 0 R /XYZ 84.259843 684.651496 0 ] (rbac-and-service-accounts) [ 815 0 R /XYZ 41.740157 784.430079 0 ] (scheduler) [ 694 0 R /XYZ 41.740157 784.430079 0 ] (secrets-management) [ 887 0 R /XYZ 41.740157 784.430079 0 ] (testing-controls-methodology) [ 304 0 R /XYZ 84.259843 166.076079 0 ] (the-default-namespace-should-not-be-used-automated) [ 891 0 R /XYZ 84.259843 193.251496 0 ] (title-block-header) [ 6 0 R /XYZ 84.259843 502.818898 0 ] (verify-that-the-rotatekubeletservercertificate-argument-is-set-to-true-automated) [ 808 0 R /XYZ 84.259843 670.970079 0 ] (worker-node-configuration-files) [ 748 0 R /XYZ 41.740157 784.430079 0 ] ] +>> +>> +>> +endobj +4 0 obj +<< +/ExtGState << +/a1.0 << +/ca 1 +>> +/A1.0 << +/CA 1 +>> +>> +/XObject << +>> +/Pattern << +>> +/Shading << +>> +/Font 1076 0 R +>> +endobj +5 0 obj +<< +/Filter /FlateDecode +/Length 2619 +>> +stream +x[Wa咪tc! "d1!v1u.׭CO{n̶II4ς?ZΗק7' 5 =jȱϯ]5ԞS-~w +DMt_C#?%oUͩGk@׃GMMɆ=럹te mGT|r´\D~6,ct3Oi='" 9ۮy-rH6я.'6Z;Ugc)յͅ'cn꿠Έx/.c.I y=NyBp]۸Sd˒"-y&?S9>y5Zk?S|HL[huݖ+:}v_؄wsےmڇy {q^7Ŏ>Pnt;3ϣeNVyei׹Kiw!ۘ͞dh&ۄ0@aF8Nq2q/_OGKSu$*=ˊ7dFirI&-dxd{b=c m_VuoV6uQ/jUh쯃f[:]! w]!ZKۃh샢MؗewfC̥wm +ƖFxSV݇-.Huo['w?x}L.t}Jv|1Nqrߜ1GwK3YGcv}tحC'!t duTne9޸U3X_E>x3[ҙݝz;Y=g.وOO:ds#ڞװeHYCϏǜzz&$u@}\YJCVmW#wߕ<ވ}&lG0UҎEbj }?o׃S ]fRJ=m *^ E:9 H +endstream +endobj +6 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 5 0 R +/Resources 4 0 R +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +7 0 obj +<< +/Filter /FlateDecode +/Length 3661 +>> +stream +x\[ܶ~_C/@Q  $Ѥ-`oq(p6-}onD{4?)?,1̗ǏK'Ȝ69=X&(7~n8Ll'5t0Lٺ㯯C: +i3't%t?Û桇HS6̖Dqp"HtIe:[;_o;7u0ho]w &xqJK:zB!qXI4ȶi7cgjQoiqCs\ 41E;;ߏQ1Br(,,q [Bt\!*5GvFt [BBt??ѽ-Dl~lC(f⦎!*p\>^C4G G[C2x Pm.=vF;Ԁ2yMف.AH#f1Dx]8;Xiڄ`jDŽh {9o-Do! чEeJMчB}Ot/-Do!z GMt8A򗌏%FoMpP fB" L%VdM] +FI;9(2lf"T@g!E&sBe4(:E&la2G{ c de3&ͬpt,$CЙ'CaYV8dVV8d8DCsA}Q8d 19Cf'&}vF0@go8,oJ8: :["qXRO]$`D#40QdgR8 HxĘ(>ptv:CxAJA0xl3"U CgcMT9hrBD M^uFGI0oaBDOqX{ZM :;.È%H*Fm Y0ZVSj5%VS@1dLjJD4:DAA +upBM:{d2CAxh5%1)9!X)#i5%fLRZMLSjJ2n!pL!$ dp:{!fC7 +AgrV)AgCgVSy2*!3?lO N$)u"y*_i5%%XLX c0Ed3Yj +<xjJPjJak5~1VS2ZM#O1`p__VS2G Cg8dLrζD"3fZM:ÊݲVSryəa)DguB""kU^홌B#,P+O(s^W{kWƄbNC6.ŋ}ߝ5K_8`|X0*TFd#te- _ ś}?/>e::W:uGxD͠vޯ-"ڀtLQw囿tsqgl3g|ͣjb|͇2|c+G4,{ʹh>awʹii|.as:ɩ+2{z=%eFDf-`ҙqYfImqָum̦>k0s3fNOީl> D|d:Zu/D9&?ٓeݞidڒVx~;9@3~b-E(oJ3g#4NRT#NOcybYt'$eg!˹?O>Hۑ| NMgxo3ksW,f߷Ϊ9OdG`Oʝ>W^(IQk{[)n5ى\ƒyY}"Aqvާ?Ў]O{ֶ}9 ۯ[sۗV^VL\/ň Eٖ atteGS /#R_їhbC ˛zX-=<.Vapԏv ta/6/R_6w.{| >WC[ILlٝoq/olS1@ٔe'^3>`<ؗp=4-o`N,Ka9̭qaMz[BYqwg1 #Iz*{^Way ]R;:+_e) +sjn9zmUv4 rCJ6f|c?71A.FSZT7\vaWnmbJ\tWw(ĝ8-/A¡X$7Ow6݁4\ahthR!4+V6T -g>T(NJl6x$Ieǀ +lg%_6ttcl@eWTK#l5O˕4U tan|7]OnZV`ݹr睄c ׁ |s~tLv3vŐ: hm{U\1򍪛7i=2__n[8-U^r[HrVljDv{-vU905%33D汶{9uuplZD_s$E8+'q#Nd8٠fqHI)\KiD9ueV6]yB[qCNώG}U9(hK hXA[G|Vsܱ =&9=4'GmN}ņNkh|Sb:J> +endobj +9 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 76.423228 671.061024 504.679134 638.361024 ] +/BS << +/W 0 +>> +/Dest (cis-v1.6-kubernetes-benchmark---rancher-v2.6-with-kubernetes-v1.18-to-v1.21) +>> +endobj +10 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 488.750423 648.111024 496.429134 633.111024 ] +/BS << +/W 0 +>> +/Dest (cis-v1.6-kubernetes-benchmark---rancher-v2.6-with-kubernetes-v1.18-to-v1.21) +>> +endobj +11 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 76.423228 626.361024 504.679134 593.661024 ] +/BS << +/W 0 +>> +/Dest (controls) +>> +endobj +12 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 485.690364 618.411024 496.429134 603.411024 ] +/BS << +/W 0 +>> +/Dest (controls) +>> +endobj +13 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 76.423228 581.661024 504.679134 548.961024 ] +/BS << +/W 0 +>> +/Dest (master-node-configuration-files) +>> +endobj +14 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 485.521907 573.711024 496.429134 558.711024 ] +/BS << +/W 0 +>> +/Dest (master-node-configuration-files) +>> +endobj +15 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 530.961024 498.679134 507.561024 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the-api-server-pod-specification-file-permissions-are-set-to-644-or-more-restrictive-automated) +>> +endobj +16 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 481.786556 510.561024 491.179134 495.561024 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the-api-server-pod-specification-file-permissions-are-set-to-644-or-more-restrictive-automated) +>> +endobj +17 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 491.361024 498.679134 467.961024 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the-api-server-pod-specification-file-ownership-is-set-to-rootroot-automated) +>> +endobj +18 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 481.786556 470.961024 491.179134 455.961024 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the-api-server-pod-specification-file-ownership-is-set-to-rootroot-automated) +>> +endobj +19 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 451.761024 498.679134 428.361024 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the-controller-manager-pod-specification-file-permissions-are-set-to-644-or-more-restrictive-automated) +>> +endobj +20 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 481.786556 431.361024 491.179134 416.361024 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the-controller-manager-pod-specification-file-permissions-are-set-to-644-or-more-restrictive-automated) +>> +endobj +21 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 412.161024 498.679134 388.761024 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the-controller-manager-pod-specification-file-ownership-is-set-to-rootroot-automated) +>> +endobj +22 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 481.786556 391.761024 491.179134 376.761024 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the-controller-manager-pod-specification-file-ownership-is-set-to-rootroot-automated) +>> +endobj +23 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 372.561024 498.679134 349.161024 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the-scheduler-pod-specification-file-permissions-are-set-to-644-or-more-restrictive-automated) +>> +endobj +24 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 481.786556 352.161024 491.179134 337.161024 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the-scheduler-pod-specification-file-permissions-are-set-to-644-or-more-restrictive-automated) +>> +endobj +25 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 332.961024 498.679134 309.561024 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the-scheduler-pod-specification-file-ownership-is-set-to-rootroot-automated) +>> +endobj +26 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 481.786556 312.561024 491.179134 297.561024 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the-scheduler-pod-specification-file-ownership-is-set-to-rootroot-automated) +>> +endobj +27 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 293.361024 498.679134 269.961024 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the-etcd-pod-specification-file-permissions-are-set-to-644-or-more-restrictive-automated) +>> +endobj +28 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 481.786556 272.961024 491.179134 257.961024 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the-etcd-pod-specification-file-permissions-are-set-to-644-or-more-restrictive-automated) +>> +endobj +29 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 253.761024 498.679134 230.361024 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the-etcd-pod-specification-file-ownership-is-set-to-rootroot-automated) +>> +endobj +30 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 481.786556 233.361024 491.179134 218.361024 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the-etcd-pod-specification-file-ownership-is-set-to-rootroot-automated) +>> +endobj +31 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 214.161024 498.679134 190.761024 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the-container-network-interface-file-permissions-are-set-to-644-or-more-restrictive-manual) +>> +endobj +32 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 481.786556 193.761024 491.179134 178.761024 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the-container-network-interface-file-permissions-are-set-to-644-or-more-restrictive-manual) +>> +endobj +33 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 174.561024 498.679134 151.161024 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the-container-network-interface-file-ownership-is-set-to-rootroot-manual) +>> +endobj +34 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 481.786556 154.161024 491.179134 139.161024 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the-container-network-interface-file-ownership-is-set-to-rootroot-manual) +>> +endobj +35 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 134.961024 498.679134 111.561024 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the-etcd-data-directory-permissions-are-set-to-700-or-more-restrictive-automated) +>> +endobj +36 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 481.786556 114.561024 491.179134 99.561024 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the-etcd-data-directory-permissions-are-set-to-700-or-more-restrictive-automated) +>> +endobj +37 0 obj +<< +/Filter /FlateDecode +/Length 3681 +>> +stream +x\[~?< AI&`ȕ%-}41 g LL S! LA@#ɢ3g ,:/&΁7\ L1yЋX1PĠ3ǐDgP A.ML@rc&J61ѻ90Μ$љ!0!ɢ3%,H CB!/:byK!0 :{HrJ6&:ʉcD(?DQ&G1b6G#.`cDgF (:P?$%(e#5ȧlƑ!r6| lA!0A lR9 R1QIHD.)"c1s`Nb`cDgJ1"$13x Rc1A :^dLSD'dKVhKR:(s 13 a@Pl,0Re7EXGd<d%0˔geqKe2ienvD>z^ӗo?}ziYEY>O;~TgL/2z^/jO_~e(ߝ 7h>>y&Ih^n]ȬK;|/?a^d\0Wge|Y7NaGuii2˧sbף?levê-aU͋.fy.GNYIܖ1qb޼wC4'_IQ?(.l2nmYc|L#߇>&nr;+ &d6s%F­1D.oNq{uDݟ0X*bM+>:Vmgզ!qkW"0cAC q8gy-y$n+O$n|kgo_?a.hMԁM֦O_ + VɍKabx4TiUHB*whنl7.C,^pbr/8e,%qVW!^{&ڳeF_yN|ɆpaYmx+tՆUUۑG[5S򧲥>MrӕoוMre8uO |KXag?w8D)ڊRߖ_>8 +M +PMv9[${fi룤>3z1uO~xG(gFYӗhًM}R {a7@~u`XlqNUMi^N-mR&j]OoP4)Wd#$4vD͕ _Bf;>./\)4\崋[ZTjO-]ꔛwda<͵8*k+yJpⴃ+TMggd60N46xmQ[ {:v0|v,vtYl%L6Sijc-uM)ڧQ]ss凕-Tkgwnǖ;Fqj zWe*la+Vϵ.Fߪ]lv$)tqOFkNn,=*TjZk)PI~o[n]JH›b҄-l!!w1 #:}O -B;: +pcs],Æb<^]nH?qTq81%<4){ +%É36*CkRSr줙^#} ;4Bw ̎MW4`89ceώ^}Q9PiKM0JT޾ݸm5κS[ECQS}t~PeNkhBŮb:jiyJ)ydGVVۭ͖MxtG;%,g7iHúZm0BW$߇{ե׏B"sNQr DV!F|}HX2~{Txm%!=o^@LWnҥ"o,A^ ;wW +endstream +endobj +38 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 37 0 R +/Resources 4 0 R +/Annots [ 39 0 R 40 0 R 41 0 R 42 0 R 43 0 R 44 0 R 45 0 R 46 0 R 47 0 R 48 0 R 49 0 R 50 0 R 51 0 R 52 0 R 53 0 R 54 0 R 55 0 R 56 0 R 57 0 R 58 0 R 59 0 R 60 0 R 61 0 R 62 0 R 63 0 R 64 0 R 65 0 R 66 0 R 67 0 R 68 0 R 69 0 R 70 0 R 71 0 R 72 0 R 73 0 R 74 0 R ] +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +39 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 790.370079 498.679134 766.970079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the-etcd-data-directory-ownership-is-set-to-etcdetcd-automated) +>> +endobj +40 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 481.786556 769.970079 491.179134 754.970079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the-etcd-data-directory-ownership-is-set-to-etcdetcd-automated) +>> +endobj +41 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 750.770079 498.679134 727.370079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the-admin.conf-file-permissions-are-set-to-644-or-more-restrictive-automated) +>> +endobj +42 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 481.786556 730.370079 491.179134 715.370079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the-admin.conf-file-permissions-are-set-to-644-or-more-restrictive-automated) +>> +endobj +43 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 711.170079 498.679134 687.770079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the-admin.conf-file-ownership-is-set-to-rootroot-automated) +>> +endobj +44 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 481.786556 690.770079 491.179134 675.770079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the-admin.conf-file-ownership-is-set-to-rootroot-automated) +>> +endobj +45 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 671.570079 498.679134 648.170079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the-scheduler.conf-file-permissions-are-set-to-644-or-more-restrictive-automated) +>> +endobj +46 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 481.786556 651.170079 491.179134 636.170079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the-scheduler.conf-file-permissions-are-set-to-644-or-more-restrictive-automated) +>> +endobj +47 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 631.970079 498.679134 608.570079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the-scheduler.conf-file-ownership-is-set-to-rootroot-automated) +>> +endobj +48 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 481.786556 611.570079 491.179134 596.570079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the-scheduler.conf-file-ownership-is-set-to-rootroot-automated) +>> +endobj +49 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 592.370079 498.679134 568.970079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the-controller-manager.conf-file-permissions-are-set-to-644-or-more-restrictive-automated) +>> +endobj +50 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 481.786556 571.970079 491.179134 556.970079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the-controller-manager.conf-file-permissions-are-set-to-644-or-more-restrictive-automated) +>> +endobj +51 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 552.770079 498.679134 529.370079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the-controller-manager.conf-file-ownership-is-set-to-rootroot-automated) +>> +endobj +52 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 481.786556 532.370079 491.179134 517.370079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the-controller-manager.conf-file-ownership-is-set-to-rootroot-automated) +>> +endobj +53 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 513.170079 498.679134 489.770079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the-kubernetes-pki-directory-and-file-ownership-is-set-to-rootroot-automated) +>> +endobj +54 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 481.786556 492.770079 491.179134 477.770079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the-kubernetes-pki-directory-and-file-ownership-is-set-to-rootroot-automated) +>> +endobj +55 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 473.570079 498.679134 450.170079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the-kubernetes-pki-certificate-file-permissions-are-set-to-644-or-more-restrictive-automated) +>> +endobj +56 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 481.786556 453.170079 491.179134 438.170079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the-kubernetes-pki-certificate-file-permissions-are-set-to-644-or-more-restrictive-automated) +>> +endobj +57 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 433.970079 498.679134 410.570079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the-kubernetes-pki-key-file-permissions-are-set-to-600-automated) +>> +endobj +58 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 481.786556 413.570079 491.179134 398.570079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the-kubernetes-pki-key-file-permissions-are-set-to-600-automated) +>> +endobj +59 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 76.423228 394.370079 504.679134 361.670079 ] +/BS << +/W 0 +>> +/Dest (api-server) +>> +endobj +60 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 481.982112 386.420079 496.429134 371.420079 ] +/BS << +/W 0 +>> +/Dest (api-server) +>> +endobj +61 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 343.670079 498.679134 320.270079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---anonymous-auth-argument-is-set-to-false-automated) +>> +endobj +62 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 481.786556 323.270079 491.179134 308.270079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---anonymous-auth-argument-is-set-to-false-automated) +>> +endobj +63 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 304.070079 498.679134 280.670079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---basic-auth-file-argument-is-not-set-automated) +>> +endobj +64 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 481.786556 298.670079 491.179134 283.670079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---basic-auth-file-argument-is-not-set-automated) +>> +endobj +65 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 264.470079 498.679134 241.070079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---token-auth-file-parameter-is-not-set-automated) +>> +endobj +66 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 481.786556 259.070079 491.179134 244.070079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---token-auth-file-parameter-is-not-set-automated) +>> +endobj +67 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 224.870079 498.679134 201.470079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---kubelet-https-argument-is-set-to-true-automated) +>> +endobj +68 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 481.786556 219.470079 491.179134 204.470079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---kubelet-https-argument-is-set-to-true-automated) +>> +endobj +69 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 185.270079 498.679134 161.870079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---kubelet-client-certificate-and---kubelet-client-key-arguments-are-set-as-appropriate-automated) +>> +endobj +70 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 481.786556 164.870079 491.179134 149.870079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---kubelet-client-certificate-and---kubelet-client-key-arguments-are-set-as-appropriate-automated) +>> +endobj +71 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 145.670079 498.679134 122.270079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---kubelet-certificate-authority-argument-is-set-as-appropriate-automated) +>> +endobj +72 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 481.786556 125.270079 491.179134 110.270079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---kubelet-certificate-authority-argument-is-set-as-appropriate-automated) +>> +endobj +73 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 106.070079 498.679134 82.670079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---authorization-mode-argument-is-not-set-to-alwaysallow-automated) +>> +endobj +74 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 481.786556 85.670079 491.179134 70.670079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---authorization-mode-argument-is-not-set-to-alwaysallow-automated) +>> +endobj +75 0 obj +<< +/Filter /FlateDecode +/Length 3492 +>> +stream +x\ێ}gIV1"m`ր~⥻53ӽ 6JR:: {S8R4'>~yq~!dWϦ>b fhsb Yz̩2|ձ-Zf@V&s|Vf`ŞQhb`UVGWK*aNҫСfK*:FBcTWg1lUL 0`UV'<0Vg.K*:!ҫ +XzUաҫ +΅ҫ +,>{Lk߰njY20e)KVFⴜ"&9u zcZPmU\|l{ar?[W/}n{&[4dwv_|q#wU]=Ow_~_>a{%Pœ S>_=,`^nɤ~oQ{kx#>=PJ7D*CXb16- edQ[n9&嶌̕7*1Q{CAM&z27ccz|GHyHLS%CƲڼy<7x|u^ "^yf#XwlG{j̥-4t8Hjo{~th6L=DV~suFn5}T~;V$CmwϏiG5̵Z-W.g(p4 Z%\SvY%洼%.Ʋˋ>P3/W/&i*!sرm}v}3ֵǺL/cx'8[_ײGsúڷ&z,hjO[;?R_GH L,uX\^Ui}ε Ϝ|sZ?N(hig?>` ˻z@ +n\_5?7?fo_z2s5 zIŴ0Ÿq8~c۫x #L\޻eֽԛ"h;&\tavaM 7Nxtn}ڴm}(Vi] twG:A6ٳ$Փǩ9^55|z5f]<)}0uwq94sgbv7a)l|n] sڍM0$aii1ڳSPcmUR" =7Jt"_?cgsyd1R9F>l&c)s(WZou<)_8ݏ[yxR]^Oݪ~  ICq\x)eluJ_(2Ek; :k2{rI-WSkS=s[z~=9Z+B}7שhA!6tO3 +endstream +endobj +76 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 75 0 R +/Resources 4 0 R +/Annots [ 77 0 R 78 0 R 79 0 R 80 0 R 81 0 R 82 0 R 83 0 R 84 0 R 85 0 R 86 0 R 87 0 R 88 0 R 89 0 R 90 0 R 91 0 R 92 0 R 93 0 R 94 0 R 95 0 R 96 0 R 97 0 R 98 0 R 99 0 R 100 0 R 101 0 R 102 0 R 103 0 R 104 0 R 105 0 R 106 0 R 107 0 R 108 0 R 109 0 R 110 0 R 111 0 R 112 0 R ] +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +77 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 790.370079 498.679134 766.970079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---authorization-mode-argument-includes-node-automated) +>> +endobj +78 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 481.786556 769.970079 491.179134 754.970079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---authorization-mode-argument-includes-node-automated) +>> +endobj +79 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 750.770079 498.679134 727.370079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---authorization-mode-argument-includes-rbac-automated) +>> +endobj +80 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 481.786556 730.370079 491.179134 715.370079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---authorization-mode-argument-includes-rbac-automated) +>> +endobj +81 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 711.170079 498.679134 687.770079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the-admission-control-plugin-eventratelimit-is-set-automated) +>> +endobj +82 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 481.786556 690.770079 491.179134 675.770079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the-admission-control-plugin-eventratelimit-is-set-automated) +>> +endobj +83 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 671.570079 498.679134 648.170079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the-admission-control-plugin-alwaysadmit-is-not-set-automated) +>> +endobj +84 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 481.786556 651.170079 491.179134 636.170079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the-admission-control-plugin-alwaysadmit-is-not-set-automated) +>> +endobj +85 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 631.970079 498.679134 608.570079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the-admission-control-plugin-alwayspullimages-is-set-manual) +>> +endobj +86 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 481.786556 611.570079 491.179134 596.570079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the-admission-control-plugin-alwayspullimages-is-set-manual) +>> +endobj +87 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 592.370079 498.679134 568.970079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the-admission-control-plugin-securitycontextdeny-is-set-if-podsecuritypolicy-is-not-used-manual) +>> +endobj +88 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 481.786556 571.970079 491.179134 556.970079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the-admission-control-plugin-securitycontextdeny-is-set-if-podsecuritypolicy-is-not-used-manual) +>> +endobj +89 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 552.770079 498.679134 529.370079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the-admission-control-plugin-serviceaccount-is-set-automated) +>> +endobj +90 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 481.786556 532.370079 491.179134 517.370079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the-admission-control-plugin-serviceaccount-is-set-automated) +>> +endobj +91 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 513.170079 498.679134 489.770079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the-admission-control-plugin-namespacelifecycle-is-set-automated) +>> +endobj +92 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 481.786556 492.770079 491.179134 477.770079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the-admission-control-plugin-namespacelifecycle-is-set-automated) +>> +endobj +93 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 473.570079 498.679134 450.170079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the-admission-control-plugin-podsecuritypolicy-is-set-automated) +>> +endobj +94 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 481.786556 453.170079 491.179134 438.170079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the-admission-control-plugin-podsecuritypolicy-is-set-automated) +>> +endobj +95 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 433.970079 498.679134 410.570079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the-admission-control-plugin-noderestriction-is-set-automated) +>> +endobj +96 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 481.786556 413.570079 491.179134 398.570079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the-admission-control-plugin-noderestriction-is-set-automated) +>> +endobj +97 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 394.370079 498.679134 370.970079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---insecure-bind-address-argument-is-not-set-automated) +>> +endobj +98 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 481.786556 373.970079 491.179134 358.970079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---insecure-bind-address-argument-is-not-set-automated) +>> +endobj +99 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 354.770079 498.679134 331.370079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---insecure-port-argument-is-set-to-0-automated) +>> +endobj +100 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 481.786556 349.370079 491.179134 334.370079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---insecure-port-argument-is-set-to-0-automated) +>> +endobj +101 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 315.170079 498.679134 291.770079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---secure-port-argument-is-not-set-to-0-automated) +>> +endobj +102 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 481.786556 309.770079 491.179134 294.770079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---secure-port-argument-is-not-set-to-0-automated) +>> +endobj +103 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 275.570079 498.679134 252.170079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---profiling-argument-is-set-to-false-automated) +>> +endobj +104 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 481.786556 270.170079 491.179134 255.170079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---profiling-argument-is-set-to-false-automated) +>> +endobj +105 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 235.970079 498.679134 212.570079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---audit-log-path-argument-is-set-automated) +>> +endobj +106 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 481.786556 230.570079 491.179134 215.570079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---audit-log-path-argument-is-set-automated) +>> +endobj +107 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 196.370079 498.679134 172.970079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---audit-log-maxage-argument-is-set-to-30-or-as-appropriate-automated) +>> +endobj +108 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 481.786556 175.970079 491.179134 160.970079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---audit-log-maxage-argument-is-set-to-30-or-as-appropriate-automated) +>> +endobj +109 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 156.770079 498.679134 133.370079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---audit-log-maxbackup-argument-is-set-to-10-or-as-appropriate-automated) +>> +endobj +110 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 481.786556 136.370079 491.179134 121.370079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---audit-log-maxbackup-argument-is-set-to-10-or-as-appropriate-automated) +>> +endobj +111 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 117.170079 498.679134 93.770079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---audit-log-maxsize-argument-is-set-to-100-or-as-appropriate-automated) +>> +endobj +112 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 481.786556 96.770079 491.179134 81.770079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---audit-log-maxsize-argument-is-set-to-100-or-as-appropriate-automated) +>> +endobj +113 0 obj +<< +/Filter /FlateDecode +/Length 3624 +>> +stream +x\݋6￢QTRUIAA}rn1Wr[R.Lf703]TOIV^~\%1O%3!'w|DΗP i:ggYqybX"( zaffaT!LH!̔$?!̈d9P PbO &z " ,bA1bN 1c6'bE "1"3 ,2#lOc&Fj`>eD7.JcQʠxfE RC BB`cX0?(u0X?(uP9JJ@mȌD:(D#2G1:(9:$P`6Fd?(u0PlLvP('Ǥ8)#Tv)JA6Fdf1:(2?$uP@`cDX<CR!B1"C6ibHꠌGƈ^2?2Βa>fLxpYܲZ43YhZG'3t?/i Hœx*rxRLZmkV ΜܳqdшIх"\@钪>4=[[XZwjPO % zU #-.yU+jQij]ߧx%}2^!lAѳ0ȵ\aPv/ʥU aEu}ޕXv޼P\xKK^(ןt>xBj KG +P|@w+t_B%D_B j[rjƼA׌Vs]ӧM XR`0098Ă,L xzH(:70QTȐ9͸>34dN>zC$N02.2Y2G2°+PPg o y"bl`Z 08%4%s.ђ9I9QR%ss&ddS4l}ySNr;9HbډK;cti'v;$w'v2I߉,9Ď8;%GNdQ;YrT؉,9*ĎHNQq'v(܉"9 +wbԇ;SCܝ)x'v=ɫ;S]ډ’Wwb՝)u;Er߉"9N;S$GN쀗$vG@ezu=Wc%_]pZo.̿_C9ob*2ޫz%~tÿ<7>W|b 糯7e@inL4mCL>94me_>=EGu=?oE7fv`>ځpg?z}[`سXo>ؑua۠yL:oTFCЈSDtd΋"L[mȭӧGx3?]俟F0t9u}?8hnk>fc~L͇L=dB2ص>o TiޜںOY ps߆ڰjSkڰ }㵰ȏO3_Yz.v:s9򚍍[%/)(}+ίq}j].9yi~ ?fXcBQ߃-0\N7'f@8_׹o1P|lˇjlR&՝;cj<;~eW5xަ8l[۝!c|I7ӽ[yEh6G]އ~LSR? z!`-,0ft )f*.oMЮ['S۫:Bv~5mҙ/7Yus+D\{_Is&:8q,5v07%("]jn%KÆ~XckՁ2ÆiOi6^RW:-4֗4PS!AiTrE7(וՂ317GŽ+ B:Vq37Jǭi~2\=ڤ*iQB\-lpE<j޹"?(i +O41*AMֺ*Sk^h hf'/ܳi0 R}VK&+t;klAU*Sivc=y,нk +kR}Bd 98{kA;u=w>jroku7i*lxI[#|v1EIRF'r֞~8l?2"RZ_p'Ҷx}~uA֕,LKtbE CXjx*_ݏސ3(HCJ3dOZ[m3 +4I3I%';rǵ8JQ +%Սrʩj bVԍtWQ"ֶԉF8(uMPu3‚1n,YugE@ީ;V=St^sⱞ꼢9#4'G5fFBEߖ15xeq^RcAYkUI[ox};;?},WhThmVm8126B>+='vV7K!o&C<:̒C))aVk#O^,^i%%1LڣO !£\(+ڥ)`k_.ViSE*y7fedb9 +endstream +endobj +114 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 113 0 R +/Resources 4 0 R +/Annots [ 115 0 R 116 0 R 117 0 R 118 0 R 119 0 R 120 0 R 121 0 R 122 0 R 123 0 R 124 0 R 125 0 R 126 0 R 127 0 R 128 0 R 129 0 R 130 0 R 131 0 R 132 0 R 133 0 R 134 0 R 135 0 R 136 0 R 137 0 R 138 0 R 139 0 R 140 0 R 141 0 R 142 0 R 143 0 R 144 0 R 145 0 R 146 0 R 147 0 R 148 0 R 149 0 R 150 0 R ] +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +115 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 790.370079 498.679134 766.970079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---request-timeout-argument-is-set-as-appropriate-automated) +>> +endobj +116 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 481.786556 769.970079 491.179134 754.970079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---request-timeout-argument-is-set-as-appropriate-automated) +>> +endobj +117 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 750.770079 498.679134 727.370079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---service-account-lookup-argument-is-set-to-true-automated) +>> +endobj +118 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 481.786556 730.370079 491.179134 715.370079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---service-account-lookup-argument-is-set-to-true-automated) +>> +endobj +119 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 711.170079 498.679134 687.770079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---service-account-key-file-argument-is-set-as-appropriate-automated) +>> +endobj +120 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 481.786556 690.770079 491.179134 675.770079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---service-account-key-file-argument-is-set-as-appropriate-automated) +>> +endobj +121 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 671.570079 498.679134 648.170079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---etcd-certfile-and---etcd-keyfile-arguments-are-set-as-appropriate-automated) +>> +endobj +122 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 481.786556 651.170079 491.179134 636.170079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---etcd-certfile-and---etcd-keyfile-arguments-are-set-as-appropriate-automated) +>> +endobj +123 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 631.970079 498.679134 608.570079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---tls-cert-file-and---tls-private-key-file-arguments-are-set-as-appropriate-automated) +>> +endobj +124 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 481.786556 611.570079 491.179134 596.570079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---tls-cert-file-and---tls-private-key-file-arguments-are-set-as-appropriate-automated) +>> +endobj +125 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 592.370079 498.679134 568.970079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---client-ca-file-argument-is-set-as-appropriate-automated) +>> +endobj +126 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 481.786556 571.970079 491.179134 556.970079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---client-ca-file-argument-is-set-as-appropriate-automated) +>> +endobj +127 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 552.770079 498.679134 529.370079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---etcd-cafile-argument-is-set-as-appropriate-automated) +>> +endobj +128 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 481.786556 532.370079 491.179134 517.370079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---etcd-cafile-argument-is-set-as-appropriate-automated) +>> +endobj +129 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 513.170079 498.679134 489.770079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---encryption-provider-config-argument-is-set-as-appropriate-automated) +>> +endobj +130 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 481.786556 492.770079 491.179134 477.770079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---encryption-provider-config-argument-is-set-as-appropriate-automated) +>> +endobj +131 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 473.570079 498.679134 450.170079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-encryption-providers-are-appropriately-configured-automated) +>> +endobj +132 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 481.786556 453.170079 491.179134 438.170079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-encryption-providers-are-appropriately-configured-automated) +>> +endobj +133 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 433.970079 498.679134 410.570079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the-api-server-only-makes-use-of-strong-cryptographic-ciphers-automated) +>> +endobj +134 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 481.786556 413.570079 491.179134 398.570079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the-api-server-only-makes-use-of-strong-cryptographic-ciphers-automated) +>> +endobj +135 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 76.423228 394.370079 504.679134 361.670079 ] +/BS << +/W 0 +>> +/Dest (controller-manager) +>> +endobj +136 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 483.326107 386.420079 496.429134 371.420079 ] +/BS << +/W 0 +>> +/Dest (controller-manager) +>> +endobj +137 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 343.670079 498.679134 320.270079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---terminated-pod-gc-threshold-argument-is-set-as-appropriate-automated) +>> +endobj +138 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 481.786556 323.270079 491.179134 308.270079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---terminated-pod-gc-threshold-argument-is-set-as-appropriate-automated) +>> +endobj +139 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 304.070079 498.679134 280.670079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---profiling-argument-is-set-to-false-automated-1) +>> +endobj +140 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 481.786556 298.670079 491.179134 283.670079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---profiling-argument-is-set-to-false-automated-1) +>> +endobj +141 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 264.470079 498.679134 241.070079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---use-service-account-credentials-argument-is-set-to-true-automated) +>> +endobj +142 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 481.786556 244.070079 491.179134 229.070079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---use-service-account-credentials-argument-is-set-to-true-automated) +>> +endobj +143 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 224.870079 498.679134 201.470079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---service-account-private-key-file-argument-is-set-as-appropriate-automated) +>> +endobj +144 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 481.786556 204.470079 491.179134 189.470079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---service-account-private-key-file-argument-is-set-as-appropriate-automated) +>> +endobj +145 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 185.270079 498.679134 161.870079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---root-ca-file-argument-is-set-as-appropriate-automated) +>> +endobj +146 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 481.786556 164.870079 491.179134 149.870079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---root-ca-file-argument-is-set-as-appropriate-automated) +>> +endobj +147 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 145.670079 498.679134 122.270079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the-rotatekubeletservercertificate-argument-is-set-to-true-automated) +>> +endobj +148 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 481.786556 125.270079 491.179134 110.270079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the-rotatekubeletservercertificate-argument-is-set-to-true-automated) +>> +endobj +149 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 106.070079 498.679134 82.670079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---bind-address-argument-is-set-to-127.0.0.1-automated) +>> +endobj +150 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 481.786556 85.670079 491.179134 70.670079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---bind-address-argument-is-set-to-127.0.0.1-automated) +>> +endobj +151 0 obj +<< +/Filter /FlateDecode +/Length 3789 +>> +stream +x][ ~_1B-@,zE6IKٖHٲ|ٴh|gw"%%j󩡿"c +.Z3I4oW|ɸ뇋Ѥ('}cPHO\~M|b ä)heְ7 +`y,+xb%՜1H &"qmf n I]?7}=`"t:lowϸJCubuqmo` %ںpNFlL.\~[]2Rɇ>Iwi''cp㯞>tlҌwFs$IȇO{W{4=xWO{ƄZ-ȸq^#\)*3Ws]#VAD\p2*a¨2(Mv$h29[#"Di Ad20CZʪ#ZJ#Z^ F\c4p +=o}6otf7ϸ( g,X64B~it& allԥ/c7#<$p8qNmN#G}TQcQ„ѸQyhG}N336q>Z?bL[L 2 Aܗ ֎d43\H1 ଆAR`ͨN+QW#Qz b +۫2*pV;੺p*;w 1w Q:@:jwj=RAQ;H5pFwjpj<;H5pFz w(-(WĹ+pSR;j>eFP*+{O5pSrTw<(8=bAl{8ş)CRs"mvBonzPwoo2.2|+g1\򕸷)Z@omoUܼ4Hݶa̞ ,- -3<5G~s݉㖮6*nNlA..[h#7S}bS^u((#qRI E_)PO3zIѓ'E?ETGݓ'EOj+iCͦ|/Fϗ̻8nʋF\\l!xۑ @_& 0iJ?g{ Խ):F-nuՌZQs0z5?E,QqQ.J8h hV7US 5Uqd"{`˪$-kX[QaS5Wq;q2qS5#nN\ \(6f'u:N1;ݧe$N=) 6iF;#:)zRGh/t{RIW-t%mB#(x$yzkv18^!)L>\o?F"廨灁wRɤ7.w_H޼swW߾!iV[Q,d({o[ +4FAKMML~5z0_ڬgc30nENh{nm ogIǓ- ̰_l弰vKle\S,,2X}93}TgvWX0?Osϭ_wǡKՂ BXSdf/1K݄PtrS"[Gp._3?Dgc&F1T۾ϗAAm2n:/eyrx͖Q:E(M~SEgd}TiyheoAߧ>]u/RTcRY"kLW/^{Fްj|cޔ8B1GC#e4ͱo_WXU m*]4βZs6?K V;\>f|W eL@:e2^3JH35uȯ=A/>@f32GL=NsP&58cs˲ܡ\jz?fQ,-:vcMх =G _ccZk\Pm6uru2}lו]\4NŐ@\]P!wΧ[K#RZ|((jwn-732;ˆG}.sa&W$_YDsx1퓔 Mr^򹷤rcJ Q<~0Z̒\Jm6s`2E"Q bP;m> +endobj +153 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 76.423228 790.370079 504.679134 757.670079 ] +/BS << +/W 0 +>> +/Dest (scheduler) +>> +endobj +154 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 481.309749 782.420079 496.429134 767.420079 ] +/BS << +/W 0 +>> +/Dest (scheduler) +>> +endobj +155 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 739.670079 498.679134 716.270079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---profiling-argument-is-set-to-false-automated-2) +>> +endobj +156 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 481.786556 734.270079 491.179134 719.270079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---profiling-argument-is-set-to-false-automated-2) +>> +endobj +157 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 700.070079 498.679134 676.670079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---bind-address-argument-is-set-to-127.0.0.1-automated-1) +>> +endobj +158 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 481.786556 679.670079 491.179134 664.670079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---bind-address-argument-is-set-to-127.0.0.1-automated-1) +>> +endobj +159 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 76.423228 660.470079 504.679134 627.770079 ] +/BS << +/W 0 +>> +/Dest (etcd-node-configuration-files) +>> +endobj +160 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 481.237972 652.520079 496.429134 637.520079 ] +/BS << +/W 0 +>> +/Dest (etcd-node-configuration-files) +>> +endobj +161 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 609.770079 498.679134 586.370079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---cert-file-and---key-file-arguments-are-set-as-appropriate-automated) +>> +endobj +162 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 481.786556 589.370079 491.179134 574.370079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---cert-file-and---key-file-arguments-are-set-as-appropriate-automated) +>> +endobj +163 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 570.170079 498.679134 546.770079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---client-cert-auth-argument-is-set-to-true-automated) +>> +endobj +164 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 481.786556 564.770079 491.179134 549.770079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---client-cert-auth-argument-is-set-to-true-automated) +>> +endobj +165 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 530.570079 498.679134 507.170079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---auto-tls-argument-is-not-set-to-true-automated) +>> +endobj +166 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 481.786556 525.170079 491.179134 510.170079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---auto-tls-argument-is-not-set-to-true-automated) +>> +endobj +167 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 490.970079 498.679134 467.570079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---peer-cert-file-and---peer-key-file-arguments-are-set-as-appropriate-automated) +>> +endobj +168 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 481.786556 470.570079 491.179134 455.570079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---peer-cert-file-and---peer-key-file-arguments-are-set-as-appropriate-automated) +>> +endobj +169 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 451.370079 498.679134 427.970079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---peer-client-cert-auth-argument-is-set-to-true-automated) +>> +endobj +170 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 481.786556 430.970079 491.179134 415.970079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---peer-client-cert-auth-argument-is-set-to-true-automated) +>> +endobj +171 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 411.770079 498.679134 388.370079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---peer-auto-tls-argument-is-not-set-to-true-automated) +>> +endobj +172 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 481.786556 406.370079 491.179134 391.370079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---peer-auto-tls-argument-is-not-set-to-true-automated) +>> +endobj +173 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 372.170079 498.679134 348.770079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-a-unique-certificate-authority-is-used-for-etcd-automated) +>> +endobj +174 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 477.090267 351.770079 491.179134 336.770079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-a-unique-certificate-authority-is-used-for-etcd-automated) +>> +endobj +175 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 76.423228 332.570079 504.679134 299.870079 ] +/BS << +/W 0 +>> +/Dest (authentication-and-authorization) +>> +endobj +176 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 477.986019 324.620079 496.429134 309.620079 ] +/BS << +/W 0 +>> +/Dest (authentication-and-authorization) +>> +endobj +177 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 281.870079 498.679134 258.470079 ] +/BS << +/W 0 +>> +/Dest (client-certificate-authentication-should-not-be-used-for-users-manual) +>> +endobj +178 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 477.090267 261.470079 491.179134 246.470079 ] +/BS << +/W 0 +>> +/Dest (client-certificate-authentication-should-not-be-used-for-users-manual) +>> +endobj +179 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 76.423228 242.270079 504.679134 209.570079 ] +/BS << +/W 0 +>> +/Dest (logging) +>> +endobj +180 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 477.506282 234.320079 496.429134 219.320079 ] +/BS << +/W 0 +>> +/Dest (logging) +>> +endobj +181 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 191.570079 498.679134 168.170079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-a-minimal-audit-policy-is-created-automated) +>> +endobj +182 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 477.090267 186.170079 491.179134 171.170079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-a-minimal-audit-policy-is-created-automated) +>> +endobj +183 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 151.970079 498.679134 128.570079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the-audit-policy-covers-key-security-concerns-manual) +>> +endobj +184 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 477.090267 146.570079 491.179134 131.570079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the-audit-policy-covers-key-security-concerns-manual) +>> +endobj +185 0 obj +<< +/Filter /FlateDecode +/Length 3766 +>> +stream +x\[~ׯslxrH m}8VqZwt.:'n#9h%7?RHÇǝ8xɿݔMcbv5$*f1rDg$@4bfȶuo':6[Ó y4uڊ܄ "*rk\Cw+)˾ȲRl{M_7+ҿimN"Dp;әpe~a.f2̿ 6t*,4Jvl.nCv> /o~qŝ6'Jm$Iv~to>c7HXV#򖐷JBﱻխ4-:!']n}#\SrؙA9R0_ޭ0tL'.F VÀs`F \0^ǀ3ygIU1]6&ipv@6! c0`,8Uô}0AɦH*ƻr4Lq!ď LV1'ĘC +#7(]*&sq m*&I.+1X&ɨ2Y{&NsR18+]L 1+=8T sD^p skpf%~|g獊blYlRR1 f3%~|Š)짜8$%JJ`˜2{bK1"5 8lUYQ1X)'%~ȃb8cP11%%(35)Cd*%`0xU1 ?gVrRIÄ +Y[ +. +,8D*Ɓ aI0xk٪gk +*& ob%B"Y8'U XyȤb8Gc&˳gV@̈́P5Q3{4gGi2G%~g TgCT'rtY6Bg2['W c#1.k ]&DdGEʀэι]3t(f (+1X [/M#dOϺB: + J5N^ɹҋ Rc{ +?嫐S8+9mI?ijed|U.hWҕ^V~db46Rw;WV-Eo)zOl+n)zK[PSݭԽ-Eo)<!^ƥ(3\vthu<|}R‰[yd)%H?\BG8Jјqfpƙ`sg<2d[Jy6k3zN 8B +d^a,8}r_ +gp >jKi(4>8S(%LPʆXxbN}46R6t% 3 M +KІVaR)F>zb<IB/-1l4U8cS8g S8ge{yqQ8czyqlbggc}8c iCyiC"&8c8c83893=i8D9sp gcXYi X2y(X>VAg"ڳ (a ߬=8~#ƃGg"oݿocy~/aq㕿7߿EG>ȳJ~o9NX -?^WC. Hnr/*"EU9*$=o X4gط<T!g*GmX&0̯r=SEr*rWxl/'sUW^qR rŜ]4{k@X+}_͝]D&{C-S\s~~F tݨ +>uP|Sp'&䪟lYh:džZ~LSߊ 4G$ p*sJLOWb9zo'K9US!5\W~e!0T->δµu%+s<[-mБئ 5:77}bHfPJa6bBY/,Ċm#\xƗ++\2{/{% +/KN4ąBaV4K@J%?IWK\mDeVv]e(hLv1Y4 H?9rPmjo{)m+6፥~˶}-"|[ Z]ԛ- RօM*k=rf,J]t^iF + [+^- +2tdK,MJ\7uv/AH;Sӛ3濧jU +0 jߗ2)F!T~ސ)ȣŒH-HMpްEmPǍ|jFlCz64~O~\j\!2ٲ| +endstream +endobj +186 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 185 0 R +/Resources 4 0 R +/Annots [ 187 0 R 188 0 R 189 0 R 190 0 R 191 0 R 192 0 R 193 0 R 194 0 R 195 0 R 196 0 R 197 0 R 198 0 R 199 0 R 200 0 R 201 0 R 202 0 R 203 0 R 204 0 R 205 0 R 206 0 R 207 0 R 208 0 R 209 0 R 210 0 R 211 0 R 212 0 R 213 0 R 214 0 R 215 0 R 216 0 R 217 0 R 218 0 R 219 0 R 220 0 R ] +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +187 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 76.423228 790.370079 504.679134 757.670079 ] +/BS << +/W 0 +>> +/Dest (worker-node-configuration-files) +>> +endobj +188 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 477.434505 782.420079 496.429134 767.420079 ] +/BS << +/W 0 +>> +/Dest (worker-node-configuration-files) +>> +endobj +189 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 739.670079 498.679134 716.270079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the-kubelet-service-file-permissions-are-set-to-644-or-more-restrictive-automated) +>> +endobj +190 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 477.090267 719.270079 491.179134 704.270079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the-kubelet-service-file-permissions-are-set-to-644-or-more-restrictive-automated) +>> +endobj +191 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 700.070079 498.679134 676.670079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the-kubelet-service-file-ownership-is-set-to-rootroot-automated) +>> +endobj +192 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 477.090267 679.670079 491.179134 664.670079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the-kubelet-service-file-ownership-is-set-to-rootroot-automated) +>> +endobj +193 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 660.470079 498.679134 637.070079 ] +/BS << +/W 0 +>> +/Dest (if-proxy-kubeconfig-file-exists-ensure-permissions-are-set-to-644-or-more-restrictive-automated) +>> +endobj +194 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 477.090267 640.070079 491.179134 625.070079 ] +/BS << +/W 0 +>> +/Dest (if-proxy-kubeconfig-file-exists-ensure-permissions-are-set-to-644-or-more-restrictive-automated) +>> +endobj +195 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 620.870079 498.679134 597.470079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the-proxy-kubeconfig-file-ownership-is-set-to-rootroot-automated) +>> +endobj +196 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 477.090267 600.470079 491.179134 585.470079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the-proxy-kubeconfig-file-ownership-is-set-to-rootroot-automated) +>> +endobj +197 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 581.270079 498.679134 557.870079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---kubeconfig-kubelet.conf-file-permissions-are-set-to-644-or-more-restrictive-automated) +>> +endobj +198 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 477.090267 560.870079 491.179134 545.870079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---kubeconfig-kubelet.conf-file-permissions-are-set-to-644-or-more-restrictive-automated) +>> +endobj +199 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 541.670079 498.679134 518.270079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---kubeconfig-kubelet.conf-file-ownership-is-set-to-rootroot-automated) +>> +endobj +200 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 477.090267 521.270079 491.179134 506.270079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---kubeconfig-kubelet.conf-file-ownership-is-set-to-rootroot-automated) +>> +endobj +201 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 502.070079 498.679134 478.670079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the-certificate-authorities-file-permissions-are-set-to-644-or-more-restrictive-automated) +>> +endobj +202 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 477.090267 481.670079 491.179134 466.670079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the-certificate-authorities-file-permissions-are-set-to-644-or-more-restrictive-automated) +>> +endobj +203 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 462.470079 498.679134 439.070079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the-client-certificate-authorities-file-ownership-is-set-to-rootroot-automated) +>> +endobj +204 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 477.090267 442.070079 491.179134 427.070079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the-client-certificate-authorities-file-ownership-is-set-to-rootroot-automated) +>> +endobj +205 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 422.870079 498.679134 399.470079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the-kubelet---config-configuration-file-has-permissions-set-to-644-or-more-restrictive-automated) +>> +endobj +206 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 477.090267 402.470079 491.179134 387.470079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the-kubelet---config-configuration-file-has-permissions-set-to-644-or-more-restrictive-automated) +>> +endobj +207 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 383.270079 498.679134 359.870079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the-kubelet---config-configuration-file-ownership-is-set-to-rootroot-automated) +>> +endobj +208 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 477.090267 362.870079 491.179134 347.870079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the-kubelet---config-configuration-file-ownership-is-set-to-rootroot-automated) +>> +endobj +209 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 76.423228 343.670079 504.679134 310.970079 ] +/BS << +/W 0 +>> +/Dest (kubelet) +>> +endobj +210 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 484.911067 335.720079 496.429134 320.720079 ] +/BS << +/W 0 +>> +/Dest (kubelet) +>> +endobj +211 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 292.970079 498.679134 269.570079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the-anonymous-auth-argument-is-set-to-false-automated) +>> +endobj +212 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 477.090267 272.570079 491.179134 257.570079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the-anonymous-auth-argument-is-set-to-false-automated) +>> +endobj +213 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 253.370079 498.679134 229.970079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---authorization-mode-argument-is-not-set-to-alwaysallow-automated-1) +>> +endobj +214 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 477.090267 232.970079 491.179134 217.970079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---authorization-mode-argument-is-not-set-to-alwaysallow-automated-1) +>> +endobj +215 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 213.770079 498.679134 190.370079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---client-ca-file-argument-is-set-as-appropriate-automated-1) +>> +endobj +216 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 477.090267 193.370079 491.179134 178.370079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---client-ca-file-argument-is-set-as-appropriate-automated-1) +>> +endobj +217 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 174.170079 498.679134 150.770079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---read-only-port-argument-is-set-to-0-automated) +>> +endobj +218 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 477.090267 168.770079 491.179134 153.770079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---read-only-port-argument-is-set-to-0-automated) +>> +endobj +219 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 134.570079 498.679134 111.170079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---streaming-connection-idle-timeout-argument-is-not-set-to-0-automated) +>> +endobj +220 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 477.090267 114.170079 491.179134 99.170079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---streaming-connection-idle-timeout-argument-is-not-set-to-0-automated) +>> +endobj +221 0 obj +<< +/Filter /FlateDecode +/Length 4001 +>> +stream +x\K$ sE @FX֝];PzTYҳ*%RHR=/Йr x|>$)7c +:jlISGET0e0 O!a h0Md$Ɖ%0D&)z9arR6sƤ +샜2!$S <)pFTxSV0>lGYVxq$ξO .h6 2sf` i)q,“s&2!DWY>LFG>B3<?= +G>BwuW37.袟 hY7kk +hbԶ'AZ,>@rlM"oU ^B[t1 ;.YU|qp#g{f?W1O6^ˡn+0 _ugabi11lZor"bamWc:VX r[o~`ꘔ)/,Gw[^ym?WYlq^*Nar#!/jw矋?޿q%F~m;1g31j_ ;ʷj]xO3]7 X5>~[=*]ԽKwu\5ߍ9YcqWxa~kV`}VK  + V4jҌĜT(+e? OjFj=T͠*ҽ F͐,`_V[nqh[gΎ9Zv{lDsxnOxxjwYQ $}o|=6~ +(>J-sļ[=lג'mj5$OTzW#WGy9^m:Ey(8pIji8_UelJd+Yɿm}n$ݰaV i qk;mf ]\6m>.m@3Q2b٩}[&lˀe -0w6U[{Bw%btVBh "h7y6՞P?@yq02O7ͫpSn,v,O-͓LѪgsL{+p"fk9jL}/nrġ,IB2q).$h8Ų|t/~ӚƄ2Ix>P~ ]hK~b S?<\;c|y~nם<Nxs{:zn]L7{"H }Gs;{t=~UiPn~ )Ɗ1a_O}-_>w*'r8|5? +*h'S>lfqH) +_dQFS?fe׍Bѭ:F'>_^w^U- +N}>B oNiX9/n;,"|/rFhW'G5fPl6Qt4ht +Y324R5Š$-MnDoxZ"q^K -֝7^6e,C0*S;юE%3}qƵ +qoa a=s.NٗOWS$F>Yi%Rvoc;P/DW5fNi8|b7}XSM=6>mi +endstream +endobj +222 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 221 0 R +/Resources 4 0 R +/Annots [ 223 0 R 224 0 R 225 0 R 226 0 R 227 0 R 228 0 R 229 0 R 230 0 R 231 0 R 232 0 R 233 0 R 234 0 R 235 0 R 236 0 R 237 0 R 238 0 R 239 0 R 240 0 R 241 0 R 242 0 R 243 0 R 244 0 R 245 0 R 246 0 R 247 0 R 248 0 R 249 0 R 250 0 R 251 0 R 252 0 R 253 0 R 254 0 R 255 0 R 256 0 R ] +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +223 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 790.370079 498.679134 766.970079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---protect-kernel-defaults-argument-is-set-to-true-automated) +>> +endobj +224 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 477.090267 769.970079 491.179134 754.970079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---protect-kernel-defaults-argument-is-set-to-true-automated) +>> +endobj +225 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 750.770079 498.679134 727.370079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---make-iptables-util-chains-argument-is-set-to-true-automated) +>> +endobj +226 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 477.090267 730.370079 491.179134 715.370079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---make-iptables-util-chains-argument-is-set-to-true-automated) +>> +endobj +227 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 711.170079 498.679134 687.770079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---hostname-override-argument-is-not-set-manual) +>> +endobj +228 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 477.090267 705.770079 491.179134 690.770079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---hostname-override-argument-is-not-set-manual) +>> +endobj +229 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 671.570079 498.679134 648.170079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---event-qps-argument-is-set-to-0-or-a-level-which-ensures-appropriate-event-capture-automated) +>> +endobj +230 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 477.090267 651.170079 491.179134 636.170079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---event-qps-argument-is-set-to-0-or-a-level-which-ensures-appropriate-event-capture-automated) +>> +endobj +231 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 631.970079 498.679134 608.570079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---tls-cert-file-and---tls-private-key-file-arguments-are-set-as-appropriate-automated-1) +>> +endobj +232 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 477.090267 611.570079 491.179134 596.570079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---tls-cert-file-and---tls-private-key-file-arguments-are-set-as-appropriate-automated-1) +>> +endobj +233 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 592.370079 498.679134 568.970079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---rotate-certificates-argument-is-not-set-to-false-automated) +>> +endobj +234 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 477.090267 571.970079 491.179134 556.970079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the---rotate-certificates-argument-is-not-set-to-false-automated) +>> +endobj +235 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 552.770079 498.679134 529.370079 ] +/BS << +/W 0 +>> +/Dest (verify-that-the-rotatekubeletservercertificate-argument-is-set-to-true-automated) +>> +endobj +236 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 477.090267 532.370079 491.179134 517.370079 ] +/BS << +/W 0 +>> +/Dest (verify-that-the-rotatekubeletservercertificate-argument-is-set-to-true-automated) +>> +endobj +237 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 513.170079 498.679134 489.770079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the-kubelet-only-makes-use-of-strong-cryptographic-ciphers-automated) +>> +endobj +238 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 477.090267 492.770079 491.179134 477.770079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the-kubelet-only-makes-use-of-strong-cryptographic-ciphers-automated) +>> +endobj +239 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 76.423228 473.570079 504.679134 440.870079 ] +/BS << +/W 0 +>> +/Dest (rbac-and-service-accounts) +>> +endobj +240 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 478.154476 465.620079 496.429134 450.620079 ] +/BS << +/W 0 +>> +/Dest (rbac-and-service-accounts) +>> +endobj +241 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 422.870079 498.679134 399.470079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the-cluster-admin-role-is-only-used-where-required-manual) +>> +endobj +242 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 477.090267 402.470079 491.179134 387.470079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the-cluster-admin-role-is-only-used-where-required-manual) +>> +endobj +243 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 383.270079 498.679134 359.870079 ] +/BS << +/W 0 +>> +/Dest (minimize-access-to-secrets-manual) +>> +endobj +244 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 477.090267 377.870079 491.179134 362.870079 ] +/BS << +/W 0 +>> +/Dest (minimize-access-to-secrets-manual) +>> +endobj +245 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 343.670079 498.679134 320.270079 ] +/BS << +/W 0 +>> +/Dest (minimize-wildcard-use-in-roles-and-clusterroles-manual) +>> +endobj +246 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 477.090267 338.270079 491.179134 323.270079 ] +/BS << +/W 0 +>> +/Dest (minimize-wildcard-use-in-roles-and-clusterroles-manual) +>> +endobj +247 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 304.070079 498.679134 280.670079 ] +/BS << +/W 0 +>> +/Dest (minimize-access-to-create-pods-manual) +>> +endobj +248 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 477.090267 298.670079 491.179134 283.670079 ] +/BS << +/W 0 +>> +/Dest (minimize-access-to-create-pods-manual) +>> +endobj +249 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 264.470079 498.679134 241.070079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-default-service-accounts-are-not-actively-used.-automated) +>> +endobj +250 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 477.090267 244.070079 491.179134 229.070079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-default-service-accounts-are-not-actively-used.-automated) +>> +endobj +251 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 224.870079 498.679134 201.470079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-service-account-tokens-are-only-mounted-where-necessary-manual) +>> +endobj +252 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 477.090267 204.470079 491.179134 189.470079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-service-account-tokens-are-only-mounted-where-necessary-manual) +>> +endobj +253 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 76.423228 185.270079 504.679134 152.570079 ] +/BS << +/W 0 +>> +/Dest (pod-security-policies) +>> +endobj +254 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 478.622493 177.320079 496.429134 162.320079 ] +/BS << +/W 0 +>> +/Dest (pod-security-policies) +>> +endobj +255 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 134.570079 498.679134 111.170079 ] +/BS << +/W 0 +>> +/Dest (minimize-the-admission-of-privileged-containers-manual) +>> +endobj +256 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 477.090267 129.170079 491.179134 114.170079 ] +/BS << +/W 0 +>> +/Dest (minimize-the-admission-of-privileged-containers-manual) +>> +endobj +257 0 obj +<< +/Filter /FlateDecode +/Length 3764 +>> +stream +x]m$>b>,K*1Đl6y%&>k8C~J=z)uճwg~ZUGU%iz{Y'Q!D|y:?iI"ѝ,/1^m8 +#DcOwA+p8E+|vyܔ9[%_~"71/[5ƉSޣ1n_$ + lj +*Y31 ~&T&2*ZG0R*F(LdxXi=A4: 7PT:3{)Qb}M(.)üۙ=*idfFR{@!pʶ(kDX搝&*NH蜟p +(qf +I=*> Q:!RdٓNE~)G*V1`f3fR˳$Ɖ%qlMpjYjbՑ3&9pcS&0=93Y8#NdiCpjmT8csfz"Cs&qxΙ`̇0z9ӱf2sL? hg93&29g"/Zs4\Uf2sfz)IO8EI`grLl"c5LSs&jك +>1p \Z&23 pfӜ39N8ETaij gĪ4 A/3¾Y)Mc$O_<N>[DK)$buyC[;L^MIw8wt}88zE/XB:lt4ehAxWa ɡו9jJH4EHnPU$mZ+U_]g Z0(;a00zEK'-]Tw& 4\$_WCO] ] ^~/ZiH. <5ChP'ۆ;\V橱{%!wC&*<+`|nTq |ף\FF2YG~v_E{9 y>~ΑmE/dõ:>Ƙq=z]kuEo{h"6vm~';r51=j92lXu^`g&ۢ\kh\hYl{X_$3tnxve?6/ڗ21};wFmi's>nުpI6^p6/S_/? G6=FgQu9*5gyb?H+/>X v-1Z}h:I>~|9'(Iщ2For>XGGu[m)_}83&.m 1eR{Pmql8604n[m=r~ehje/bZ+:l~e,|y9zq[>orvRy0۫jꁶd9c~50ʎ6 + e%KET:{IczbM: Yrc uAgJd[h#꺹M[߶Q[&~٭D4钒D-7N:6H݌N,(jRUG6B\" +Ӽ.W{9j{O)<|e<*Hhx5u"r=kd(G]-Ȟ2Vj^2( [<|HꇏV4A.Y{Lvȗ_ 0&Z6\KZˆX47j4@Wu]{ANRC;%NX&}Z ӷDS0;FG3Z\ A?Wl Px2gyT~Vקq,db5Pt=b~dX>um wMb$-M3JoT.prU+©֝+$PP6j +b3Av!l mA-5c0Yrn.]gEs^N֮! +bP bV+JEـ42s|Tb@P.F)_B47SP:~->(Gqj-=\ l) x"?+4'A+ڭpm4ԇ*͂5UPESY7]4 Ϛ$qWZ)M, }b:f e%C VEo Mhu@kl؋O%H͇:t_(h xYyNd^M+S[:48Ndw4l+g$%VSq!FT(Ͳ4 d?J*l%N4"2+. !F\kT;{F8^{6ٕHhM}hXBkv9ui9vs%xj[VĞmj݉ ,@R)SZ +EM}ˊ[|S4'iT'dmdSْ ٷ&Ya9yuEIڂmucT۳6C\vm$_^;[FRV u C?R`{i8y bo7K["eDjb:xn7*`>-)'ut " +endstream +endobj +258 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 257 0 R +/Resources 4 0 R +/Annots [ 259 0 R 260 0 R 261 0 R 262 0 R 263 0 R 264 0 R 265 0 R 266 0 R 267 0 R 268 0 R 269 0 R 270 0 R 271 0 R 272 0 R 273 0 R 274 0 R 275 0 R 276 0 R 277 0 R 278 0 R 279 0 R 280 0 R 281 0 R 282 0 R 283 0 R 284 0 R 285 0 R 286 0 R 287 0 R 288 0 R 289 0 R 290 0 R ] +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +259 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 790.370079 498.679134 766.970079 ] +/BS << +/W 0 +>> +/Dest (minimize-the-admission-of-containers-wishing-to-share-the-host-process-id-namespace-automated) +>> +endobj +260 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 477.090267 769.970079 491.179134 754.970079 ] +/BS << +/W 0 +>> +/Dest (minimize-the-admission-of-containers-wishing-to-share-the-host-process-id-namespace-automated) +>> +endobj +261 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 750.770079 498.679134 727.370079 ] +/BS << +/W 0 +>> +/Dest (minimize-the-admission-of-containers-wishing-to-share-the-host-ipc-namespace-automated) +>> +endobj +262 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 477.090267 730.370079 491.179134 715.370079 ] +/BS << +/W 0 +>> +/Dest (minimize-the-admission-of-containers-wishing-to-share-the-host-ipc-namespace-automated) +>> +endobj +263 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 711.170079 498.679134 687.770079 ] +/BS << +/W 0 +>> +/Dest (minimize-the-admission-of-containers-wishing-to-share-the-host-network-namespace-automated) +>> +endobj +264 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 477.090267 690.770079 491.179134 675.770079 ] +/BS << +/W 0 +>> +/Dest (minimize-the-admission-of-containers-wishing-to-share-the-host-network-namespace-automated) +>> +endobj +265 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 671.570079 498.679134 648.170079 ] +/BS << +/W 0 +>> +/Dest (minimize-the-admission-of-containers-with-allowprivilegeescalation-automated) +>> +endobj +266 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 477.090267 651.170079 491.179134 636.170079 ] +/BS << +/W 0 +>> +/Dest (minimize-the-admission-of-containers-with-allowprivilegeescalation-automated) +>> +endobj +267 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 631.970079 498.679134 608.570079 ] +/BS << +/W 0 +>> +/Dest (minimize-the-admission-of-root-containers-manual) +>> +endobj +268 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 477.090267 626.570079 491.179134 611.570079 ] +/BS << +/W 0 +>> +/Dest (minimize-the-admission-of-root-containers-manual) +>> +endobj +269 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 592.370079 498.679134 568.970079 ] +/BS << +/W 0 +>> +/Dest (minimize-the-admission-of-containers-with-the-net_raw-capability-manual) +>> +endobj +270 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 477.090267 571.970079 491.179134 556.970079 ] +/BS << +/W 0 +>> +/Dest (minimize-the-admission-of-containers-with-the-net_raw-capability-manual) +>> +endobj +271 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 552.770079 498.679134 529.370079 ] +/BS << +/W 0 +>> +/Dest (minimize-the-admission-of-containers-with-added-capabilities-manual) +>> +endobj +272 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 477.090267 532.370079 491.179134 517.370079 ] +/BS << +/W 0 +>> +/Dest (minimize-the-admission-of-containers-with-added-capabilities-manual) +>> +endobj +273 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 513.170079 498.679134 489.770079 ] +/BS << +/W 0 +>> +/Dest (minimize-the-admission-of-containers-with-capabilities-assigned-manual) +>> +endobj +274 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 477.090267 492.770079 491.179134 477.770079 ] +/BS << +/W 0 +>> +/Dest (minimize-the-admission-of-containers-with-capabilities-assigned-manual) +>> +endobj +275 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 76.423228 473.570079 504.679134 440.870079 ] +/BS << +/W 0 +>> +/Dest (network-policies-and-cni) +>> +endobj +276 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 478.070980 465.620079 496.429134 450.620079 ] +/BS << +/W 0 +>> +/Dest (network-policies-and-cni) +>> +endobj +277 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 422.870079 498.679134 399.470079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the-cni-in-use-supports-network-policies-manual) +>> +endobj +278 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 477.090267 417.470079 491.179134 402.470079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the-cni-in-use-supports-network-policies-manual) +>> +endobj +279 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 383.270079 498.679134 359.870079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-all-namespaces-have-network-policies-defined-automated) +>> +endobj +280 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 477.090267 362.870079 491.179134 347.870079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-all-namespaces-have-network-policies-defined-automated) +>> +endobj +281 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 76.423228 343.670079 504.679134 310.970079 ] +/BS << +/W 0 +>> +/Dest (secrets-management) +>> +endobj +282 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 478.118587 335.720079 496.429134 320.720079 ] +/BS << +/W 0 +>> +/Dest (secrets-management) +>> +endobj +283 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 292.970079 498.679134 269.570079 ] +/BS << +/W 0 +>> +/Dest (prefer-using-secrets-as-files-over-secrets-as-environment-variables-manual) +>> +endobj +284 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 477.090267 272.570079 491.179134 257.570079 ] +/BS << +/W 0 +>> +/Dest (prefer-using-secrets-as-files-over-secrets-as-environment-variables-manual) +>> +endobj +285 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 253.370079 498.679134 229.970079 ] +/BS << +/W 0 +>> +/Dest (consider-external-secret-storage-manual) +>> +endobj +286 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 477.090267 247.970079 491.179134 232.970079 ] +/BS << +/W 0 +>> +/Dest (consider-external-secret-storage-manual) +>> +endobj +287 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 76.423228 213.770079 504.679134 181.070079 ] +/BS << +/W 0 +>> +/Dest (extensible-admission-control) +>> +endobj +288 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 478.131038 205.820079 496.429134 190.820079 ] +/BS << +/W 0 +>> +/Dest (extensible-admission-control) +>> +endobj +289 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 163.070079 498.679134 139.670079 ] +/BS << +/W 0 +>> +/Dest (configure-image-provenance-using-imagepolicywebhook-admission-controller-manual) +>> +endobj +290 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 477.090267 142.670079 491.179134 127.670079 ] +/BS << +/W 0 +>> +/Dest (configure-image-provenance-using-imagepolicywebhook-admission-controller-manual) +>> +endobj +291 0 obj +<< +/Filter /FlateDecode +/Length 2232 +>> +stream +xZ݋$7q,r $w= G&#,YU]y_ɒ/\ρd&璢ӻɚ,4͠>7&4ņ8lbn~;%e_ '73OMSbԔd \1\&McDS+(Zœ8X!q&,a>cI Yf`5 I8ڭJt|Nǿ}:>]HaKNǁj'!3o74(hq Mѳs'4 &33}(ڼMPIx MLbyMH9b>q3]3}Pz*h.tS V ghbr9\r6M6Nc&݉yyYc&݉yQ<;1r;1OK2N;bw'uU@5ڔδvvK<s&/%*w22Vq`~x;}|?_N Wa-\@j ܝoBMA#;V{>o0:7vv]SmK{oQv?,jtx؀Uuͫžj|EԦڦu]8/ }- |$>2 Fsgx<[^xΏuq}Ƀ.vIvl0QkT3ڝ[C{TLjMݛz㎦K-!lrR%6TZ]Yzhq GezJW+&-ެx3sT_a͵ ޺>^u r}PM`,58G>~LŌ'hNPd!A_{!zҧF&Wg9?n'\~i&lc+JWs%^Mx[K *p]}=[arBQU聰8zM;wapVaZ%niQ.٥6(3-&3pHF/(`E+SJASCb;sˬ=9S+d3t2unA<*-?T^6ncF2qQC/p`_۬w2պ/SHa +84_2ДTóR]ԣD`z4xk$G:ӠU nlkjU&Sx*^-RR<0H \KGj-+A$/x$U)YXA8Jh8!zYyk~f!^B)p(FEa^* JˑdG;2:m]R_9=7iJ@Xd]x: J(*]PQ)^4* cԵaeqsp /rI3Rc~31> +endobj +293 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 76.423228 790.370079 504.679134 757.670079 ] +/BS << +/W 0 +>> +/Dest (general-policies) +>> +endobj +294 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 477.986019 782.420079 496.429134 767.420079 ] +/BS << +/W 0 +>> +/Dest (general-policies) +>> +endobj +295 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 739.670079 498.679134 716.270079 ] +/BS << +/W 0 +>> +/Dest (create-administrative-boundaries-between-resources-using-namespaces-manual) +>> +endobj +296 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 477.090267 719.270079 491.179134 704.270079 ] +/BS << +/W 0 +>> +/Dest (create-administrative-boundaries-between-resources-using-namespaces-manual) +>> +endobj +297 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 700.070079 498.679134 676.670079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the-seccomp-profile-is-set-to-dockerdefault-in-your-pod-definitions-manual) +>> +endobj +298 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 477.090267 679.670079 491.179134 664.670079 ] +/BS << +/W 0 +>> +/Dest (ensure-that-the-seccomp-profile-is-set-to-dockerdefault-in-your-pod-definitions-manual) +>> +endobj +299 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 660.470079 498.679134 637.070079 ] +/BS << +/W 0 +>> +/Dest (apply-security-context-to-your-pods-and-containers-manual) +>> +endobj +300 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 477.090267 655.070079 491.179134 640.070079 ] +/BS << +/W 0 +>> +/Dest (apply-security-context-to-your-pods-and-containers-manual) +>> +endobj +301 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.673228 620.870079 498.679134 597.470079 ] +/BS << +/W 0 +>> +/Dest (the-default-namespace-should-not-be-used-automated) +>> +endobj +302 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 477.090267 615.470079 491.179134 600.470079 ] +/BS << +/W 0 +>> +/Dest (the-default-namespace-should-not-be-used-automated) +>> +endobj +303 0 obj +<< +/Filter /FlateDecode +/Length 4723 +>> +stream +x\ێH}g"!E$ B<ۻB=H >'mgf}.rD:y"2첾*|/9=cpˏ5Eo1oU+'sHlb&mrx)`rv)F +%,)XPGgܞ3p$JPú힕1G +zhA5`xFO!e)/aStVƳƔj5eu[e5ybLt]|&@q LK.10)O.F`d?#4xF3 +xX ]V.Ak 0,gXC0,[fX`NpbX`6ǰrJaZ`' r ,P`Xu)0,]bXLʃ.`@d8*2,p],#ð^abe2))kr pDʋ֠eXc +,swWagQB`jxஶԌ&+3,@8LI+.Ix.`0,@TP^HaxRC.{fX[˰ wCCy&ðu xXdMXR0pWixJ̰ wKdXb!İ +N Âܔc BV䣼03, +ȢM(>yJHdYƃuy r(qb[Kn_X:<9qa7.9-Fe~ߩuy r޼y~\ɠG-w]S9,뎉dsA.s|:.zXh?̇ ;.1~0<qa_\5ay!ox3. ދ凑VE__0;{nyGKҥ_xqa_\5ayš`zg>\Y>b{{t!5 w7iœQoёe x-%3;.$W _.W;J ~~\{\7h+{<̎ ;{_,s;FN~~\{\7h{<̎ ;{_,~󀾟> Zr;N{ 'd\Vt՟~~|JߛR6i{KW7/4i<~VW2ulo9eV_W9^ZVʹ~C6+f;q-o9[~hVuàWSȶx2'nyX9;sm rױvM]w]}7`JQ<0Y/}~z_UPn|W-zΦ OW. r>SԷx=zϸ|͔$IznRܞgK+8yxv}lUEb뽺:,l" 1?ڋ?4yr^5伵r[o㓈ǦjlYL* +Sއm%PvVKbVK'Em0\+ c$u\ ڧa$.ɾ@zTbApMn'uygKӷ1_9-~dZĮ@;Z:P֏}{pӜޔ;2 +2Yei7][qZYf ۲DwN}1]?]u`#uCPݸ ٴ5 +mcVȼ7=a/Nm)U  SٞB$ОuF)₹k6YR^+l8eD9z+m)iy;GZ2Zg7#mkl{p]+SOo¿Ke ѩ% +h_RW]9;"W"wLZuy;W|!D׻ӲSz6a,[n[T4A#o +endstream +endobj +304 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 303 0 R +/Resources 4 0 R +/Annots [ 305 0 R 306 0 R 307 0 R ] +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +305 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 84.259843 740.570079 409.271317 725.570079 ] +/BS << +/W 0 +>> +/A << +/Type /Action +/S /URI +/URI (https://releases.rancher.com/documents/security/2.6/Rancher_v2-6_CIS_v1-6_Benchmark_Assessment.pdf) +>> +>> +endobj +306 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 351.653153 216.824079 461.230790 201.824079 ] +/BS << +/W 0 +>> +/A << +/Type /Action +/S /URI +/URI (https://www.cisecurity.org/benchmark/kubernetes/) +>> +>> +endobj +307 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 84.259843 201.824079 165.615800 186.824079 ] +/BS << +/W 0 +>> +/A << +/Type /Action +/S /URI +/URI (https://www.cisecurity.org/benchmark/kubernetes/) +>> +>> +endobj +308 0 obj +<< +/Filter /FlateDecode +/Length 3498 +>> +stream +x[Kϯ$2p 6Q9|n35|!ZԬXV8{|Wjm%]IzCoI๸Z^T\i)|MSw|vqL8GZ!y',&Wj)>dRrĔ V[lbAT8=@ n5Y,1VU!q tj# b +rɃ e(=[\-6W %Z<" ,^s9T-C|x -daeBɃ,xDh +Y]A}x\j5|-}ryKM2y()!K<ȒGGcJFFn,^,16yƷ fK , 8n-W +B ĖV, %qdaDɃ,,, ,2ܲo޲BLoaqfpfas iatRlawDi PR"f +.-,`n-,`n,H&8 YXݘ+swcF8xB]@E‚ ψi e p7T5 +4n-B‚E" pY-,;dhaABx wQ7$ .5yV X ;_$&88R`YXQ/˫+-VS3ʿ Fy槝h/'＝x~;{z;oöX?b:5w{'w9'{|oi} +Z2l ȿ%xC==Ēy䙒'M4EV撹ZǼe_(5G]nɡ#<y4V!GJ{ߡN00rzLwnO<~-p\ūSs esC n\(ZA4B|b){_)xvfE;;`9SM9[vcY+-Pr]:?L&X8`gGZ)4ʝ.9,WL/šZǼ0)io91M sߥP,6RP0H*c-7 >A78R93:Jdy}^c?|:,u |i,)p6ڥ>7^(w{e{嗏mF$&r%=Ӵ4{#Lqrީ](2^KH̖?)NY6NF6ֹ!u +T\(9p,ܻQ*ܙIA׎ڜMk +إ 󀴵l!wVIX6$W7e_ݟ է$&y("O^b[4?4~a?^n=*u)(.pTC{lNՅ(߸3;FC'L+6_~|? / UcM(B<\;XI>b̼G?0i.m=As؍Oc\_9o D,d8:[y5!k5Er ry#ئ8Dת o,ʜ/(}~jR哜1.砑Mj=Ax ]Ns?fsFÑxkA~liZ^u mCKT E"_4,UM#^Ng$.;-i&w_U!4M^ TIKɫZHlNQI'갥eU +L +tg况ULK HM8IX~!Г+_%g\,iλybU˴xS9AƷdq:Yg[) Fl︚Z*XԟTbVw}eޑfDJڝڝ_=g-Y Z֣'WOȶg ^U5,$1/ޭ^߄ BP(R^ a^pUW#yw%7"Ds8&%۳E^f鳇5on)'VkוBh6iU./~kү>3]B +endstream +endobj +309 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 308 0 R +/Resources 4 0 R +/Annots [ 310 0 R 311 0 R ] +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +310 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 84.259843 739.370079 128.273270 724.370079 ] +/BS << +/W 0 +>> +/A << +/Type /Action +/S /URI +/URI (https://kubernetes.io/docs/tasks/tools/) +>> +>> +endobj +311 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 345.193925 739.370079 356.281327 724.370079 ] +/BS << +/W 0 +>> +/A << +/Type /Action +/S /URI +/URI (https://stedolan.github.io/jq/) +>> +>> +endobj +312 0 obj +<< +/Filter /FlateDecode +/Length 3610 +>> +stream +x\ߋ$~b 'K/0C&Z\1[\GqoV]PHH3QX\!1 rlqA3eͅq +s w}V 5Z\-x7-.H]  (,.@~suMYsjqAW0)+,.;: EzodODdg= P ak4U#’w} +[2Y\6-.EX-.Up WvlqAn2|,;' @Y pa PXFZ\P"t-.(dkշ=.ps`2W%e`UKtm*dX\P +;`qR"-.;),.4PՈܒ"T`qAՠ5V2{Iڒ}frvR*Z\0NLiʠ hl`m).Zl7BMo .,.h>;)-.C-T7#[\f~[<#neo CŔAM͈[p`'KKllqAYo7/ Torx750%2MTރxSA}f 8yzry{y|ed^4Yl^5@Z0x'ܮ>KTP3kI6{ٌhv>/ ˕#$HAXذO0k+*Lrb/'c[ +v-`]p5ZЋ'l#)Ͷc8 ws&WIϡu,^Na?>5A*iM`8N,mu@vY]8bخ5Z[/ZekUs,Q²}mn|h[8-c yjPU>Tz<ێ4}(>499ԴeK*+w>~1Xvc1v@d)ۿK}fr>~yDiՌ!6|/| >3Wr/xY=r.E` ?ϯcLSvjBkːv2dm߭۰ }A!ʎ8 1f;nڵГՓW+uOѥ7k}׶rߗMr3|ڦlZ[&ﭷ+>‚`bޯ#,-lцwVQ}qJr~𛛬ja6Vn%yeeh^M76mp0bN7U~2%y؏nEՓVi]9_#f9mBs{966NZquێ;ւnLkĉD*++g(sum.;[[z譎o~yoHOvLo9 ݗI`o-[X]ױڎtw븱~ۖ`3]Zu49si3(1Bx8˲=<&;Vَ2fg[}tjuu8sGn٧72~|sy lC'cߟwIao|&|W6]e˒}7V8bgKkՊ^Z6$w?=yW ّC;˳ULa1l8q +V/CfWe߃W$&|= ؀',Տ8 vH4ٵuR 5>9?-Ϧ6@{VXޢQGyru<@^z5܄#w}u ~\M9 C(m +z:8SF3%U83CQ)ՀmjQSR2L/^ +ԁ*#䐦g]wljm1z*jUcb³QZp:AiأQИo*I Š `cui,ˌi2j- #&*VoTK4 +ݹrԒ>{<[Rǎcvmj<ɳ2thQ1c>`?˕P+9ל!,/)\եG둰| ~O[+O)=w|h#,/x[X."[Ey#I[5TG\>Y~r +endstream +endobj +313 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 312 0 R +/Resources 4 0 R +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +314 0 obj +<< +/Filter /FlateDecode +/Length 3664 +>> +stream +x\Mϯ,V 0 @A4VݵҮ[XͰM> EȕRsɻ=-56E~s<"T +8\c|\/N^CjM7v죡PybC9d9 VK1K*-`HP`GUI,`PP +ޛ:" E`dTj.CG9X:fP$KW]J)K''K%t"\K‰P2[ ,֩ :`ɓb-"BK.G]-M6uXQ ]c5uK"tȹc],]:`[ , + +rK  NBNrfBNR'YDX$ ErMp 8 *X\ ,A +݊dqoSA-.`0.$0(T-.`p.;-.`.{SA,%Y\]xW:`[\vtL-..x(X\Yudq|ŅR ŋ !e5x7D }@AB?h1X\T5 rTj w`L]uaqcʎRD.[\bq8! "xҁw)ToqA"o!͑뚺|-- /`S,.;<Ex$$=X @eki$0z [:Z\6-.HYX\Yp q5dqA:|.9CAYX$H]Z,.X8 r  #p}`d\"䘰U-] +dе+Q gqdqR`[\w.Y\БX\+`X\PJ@fnXd[\L:" JdPLDh%ajK'^"ihq v`JS, 0?َ`&9X8n JMof d|TơJ `]~X\Pc-n+'of KDbIS5-8 MKX\PSfi#lwc3Y:.2dq +o"33[YtYꍈ摁"Ze|ud&v%P7 <8z4Q%CF,YY,}sj@?|{?/Cp~kd rK{&HSQZ\$B1m=/ Ċ,#M*抓Xf r=LɂXdl7zͷ3|ywWzLA?4^.;ˈy~~(&ykmMr"MxG~_kwimE^zx߲i}Zڧ_Ytrų}p :Kr0d7iHPjh[|սPUaewܴd߭m=:Umzi2Ode68S۴4 ~L۸_\\q}X}bgQS( 93P~A_igZ?|vJE*Gk;WO7;\am;ˬiV쭿FGF|">:PekGg~:,N6:iXrkftNcm_qI Y'ĶﶳE>;m=Ol `$pߎ->mlixB3c;-=dcoD󔯿~~ qح۬k>n;I#Dc;&>;mG=>eͯ'LoFL[&`(1o?"űO)~x)m#E=~O#EѕۑiDHGFߎ~ڍQtvs7kۗ];~zSθR[Yv~+ީf zw2+!Yj1M>v~=y2pˆ#@8mC7}󯩝m~1sZZslեvy{CrƕzmhS~hun٢Eh1v?\Hr#(THT)Mw1?'\o^ak)/;zcǛäP 7/2npj43L9ruҖz̷Qn̖^7 +´wz%|CYXD9^';)rar.R,r7՛#6Wž,ļ]~Hu?sOy*fn=?mI_=Ayi˱89H<@ Zt#t| +=Rs,UaYX5Ckm'Vͤ=3~3h{Y -۹:nw׾wmKnYy,~˪&Y< iŰt~=wj׷I{SanXR[kH~ +C| xDm ui &"_RUv>(UkdžW&:u*W㵔:fG2 IB?u7AGvhV7i:o /'t{)ZSͺwOjVOy2D!|[ӂi1쫰4G^zr2뭭>S*,? W o.SnF<[nS\2M8/(}%)fv\csYS]\!ͥ +K!VJKy/$ݥAQ⭑fy#H;r>;ۋ7+bdIk$c+u]ӹr q|/{g&w!N;gq3NVA۝| /K`X]N8Fgp) EVkg'}S}̊O%A؉I_,:uNr=}w[79+=ue3 L?TFؼ&qjRgPY 'q'Ί=y܌}-ra}Baex$JZuth%L07a60!NSG)saJ#C'#M72 +R` sBsuy_!TکKgiJ;R^i[1ZhwX䌥1R fe\VsdN[tobnzY32bZ)bH-YFC]mE-qOyΏ=,,WwdǾewi2qY<|=e/XrBeDŽ@-9i|I "vq2|5J'"d g?2dC>jȾzsHf__YZBuT5-.|+L1P,?CvƏ +endstream +endobj +315 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 314 0 R +/Resources 4 0 R +/Annots [ 316 0 R ] +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +316 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 205.473057 101.433071 186.723057 ] +/BS << +/W 0 +>> +/Dest (cb1-1) +>> +endobj +317 0 obj +<< +/Filter /FlateDecode +/Length 3947 +>> +stream +x\[d ~_UU I &<0yϧ:u;}4=dClRI:=~>/59|ˉL +o +gKL%pY>TKtWk7cΔ _{rJVWX!\ ++Z% FHJK bsYae9Yde1l IaY<B)<Lңs\\Qy <6XWL1dǐ%/@`IA9iY| kYS^, ExJfVy<)y {x֚*MI9)Y'9;)79CGNئԛIÂB4Y8xDQo:ְ~)fְxBOԛI% EQyISQz"dq5,(1J5,@zz$Lqp7 "3&2['oHtf3' {zf@_C2GaLKӧ*bfN`d3Ͷ_<}{ǏIY0~ϱb_7&m'H$GHe +iMD(F-E"E2SWƻyQ*X앏od:B݋/ {_wyuޫ;R'9=ռTEY;$2mz] +ڼ}^y;G yv# ":d 'o~>}?~ş1} +=N|-qeNmsWC^~CC&6zl4eч־}\ۖ趍J}kjǩzoǐ'4ioȢ.nu۸xSˏ(L ,[,fo<9"V58)QAwNnև!/*Zv87/YCc+id^7?;MAr$Se_= =,,| _Tf/AAO7Sn^}Z7S˼[.YcxVM -(@`rq3JIZT[{ѻvY|vzXKfw+zqku֎.sn wE>nhvYmk՚=*ʫ-RWf{s깲+mHFL-ՖGK>6-Qr va hGЏ ~л}8c7{X[q}Dr;Ďp0Ǻw{o]`7,w{Y-ui{Sןfj1tTT^/`_TZgu.{efg7)fޯC.ʼnos6[1aK֚~o2\jQ(ّTL[Plo')>9ӽi*=t/ibO {՚2궕D=e[̏R)rjҚZ>cAN5C/b|x>L n0=\L󋒜vXRn_ݦ6ߗ,yrB-khJT^f\r$6Մ7-n.e_2|hv)޲\& %f,21L *;a/H;vY.Kv{Ds]0ƴ*O떤Tu/nEq+x8_Q~/腢.k^WE{qu=y7h G{7ƴýܮ^2m}i/2}5=ul{1cmca9^6ezۇۺMؽS56ۆc]w+n'_g؆y!'ޯ"a`roߔR.;cCBzg搣$OGv#9Ov>Î"fK%;^_GR@PoiynXM^0rzc@ )|vNU}P e[|~Ws r])3;ƿ} mb},WS(fczl@g۷V3=V%` 墵n(!^4pt6Y!3Wc!qW 7rFv.HлCLn+DHՓ,/fU8pv|l=-p$2$;XyIb.N}U]ui[I>Tyծյh"T[ O +rRS욼s7$Jo.']-JU]KeC +%y,c)/v)I~:+P#/ri/rW\qo0mu\Q\oQHw|~?EzT!2sGUO)h0adž נ:@Ki gzcz|;t |w?|_7O?Mo|瘯KZ_o&%V"Sm_77c襦{o¶*XzuW_1D+:ʰ0^6f< "*݀?.ٻ 8-nOA/DڻH{[DؓˌiN*EoP@IY1zDn#56WKtc^k%4IW_v\y^cklǩ[Yc\읯% {Xwq|Z+|cʺ=mʫ+X9OS43rXh}Y)iks?8~(bw:mqK2/\h\~aHBFߋi3><.cqgu|t=|=f";i-Z/{-CfZs֘5xˤif1KF}[Nʡ/ych-iDWnuϮyrV ftOE~^=>Ǿ?m*JS~$S R;VDwA8BcN~^}>Ǿk.,Vcƺ.6^&M|~ +WԈ2HG2F2g)[#Q甉5nqs;1T+?_~wyyv2ı9}r8ig\x QoI(\x!y>NӞ8Ӫ5og`O/+c]mPv+72^uɶi} #]H)\ƻKݴBqg3:flp<')烾چcz'% 6M~q?sBj}G}dtLdrq}Z%!;\bEFXG}D]NPSݷh8}z1v5z~lsvB/O]683X_N\qnJLfG2{IB Pwץ!Ϯ!Or°ޙR}uռx2vil-H'Ct9ϩu-̮fMg +K $jkg.'[3{2NÔ\ e*̳;q+5)nAx9:VPl7 y귋 x\4L]Y>`u0AtmvI߬|98,C#%.qMGJVt"Nq8 Xe35 A~u`*#NFn*&(y>;?Z7Fz&Tޥw_J;l)/i{s-Ok{3(-ds0$-㴵A& EddlJi-KC8Jwbg%j-j{phd1di>Ke&;-۱[=O!Ð4*SC߇9V eD->swPI }M7Kn9+9iJ0N){m= wPSW7اU2/pUG Z~]rCf*oga-zT~; +endstream +endobj +323 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 322 0 R +/Resources 4 0 R +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +324 0 obj +<< +/Filter /FlateDecode +/Length 4631 +>> +stream +x\[$m~_V+ 0XI~00,0Ӟ C~>Jfw9sAOwPR]-s/39zŚ,#>)\d+1/BdR >./?:M)ѻpWa1SAa#Iɇ3b) bĞY bpAcA >+ 1{V>9XÕykU &BQygb.K^1b %8f SN/BdUd9KC;Ee*h+dB*gs%ų>YCH_Xdxi@@, Uy +kX@D3iX@@[hX@[iX@[.jX@\i5, . T4, `.;5, .=iX@@] +kX@]5, .0p5UdkXt42iX]r6iX]װj`Q؊9[ e BK>H0p A]װ}԰b?Va=x4,@c#-Ѱ w]AS|he\.QÂuӧq WZ6.ƢaA"P  "װ w!RkcXKB3TQ V@JY`~d4,M,sTde5,x5,mz3iXP,dAƧaAHa`|@)Opnwk9/e1c׺PPe*]ii:ijRv¶>3V*NR_y]I՞>rI< knù'[y~jȹcÍxMkꛧNLk]7{N޺ SIWh15۹קrP\oaO~Yk?w>i~8q5]F}n~1cT\s5԰nE}:_v91G~Xm{? Z,u-J]|]c~{[ֱ|]y/B֯oFU~-ETBPڷkO)cex[bk1 '`oۺjТ(T%Ylɖrawh??X5skNw~k$jx x +i +D_n(J,K\6#pu~(tyCLW/ʻ&,C-z]UNXV@5.9#mqIEȑҲVW.W:@,]:F@̀`ZrI.qǙR9b:H " r4uqQLD>nG{x#s=<Ԑ,-aN0ME6) [6n96l;Ek +TNGq <Hsu>6hnr'{ rw]=ΧLXK6*%9B9OӢ5YJ4S8\Ꝭ&ZپE_m(*uarH\ܮyq ÒZ;.%r7ڃ},TS"`V]g?U(c_MXy#z^<958o/:\y4#у,lk0r[-vA$"ZV'-7^$Ó?>x,R_Gqa"cs.&z˜1ˢ-Jm> ʍP\g &[QF(6:۪0[tJy·QNf1;1?Ol=#Oǃe]-EPV躺]&G Ѥ +<5=%Oq'vҍwdiXߵ\O^OKNO=ʷܴNNlNJzi_W`,HKd6\ l q&WjiK Yy݆> +endobj +326 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 507.723057 101.433071 488.973057 ] +/BS << +/W 0 +>> +/Dest (cb9-1) +>> +endobj +327 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 488.776036 99.933071 470.026036 ] +/BS << +/W 0 +>> +/Dest (cb9-2) +>> +endobj +328 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 469.829014 99.933071 451.079014 ] +/BS << +/W 0 +>> +/Dest (cb9-3) +>> +endobj +329 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 431.934971 99.933071 413.184971 ] +/BS << +/W 0 +>> +/Dest (cb9-4) +>> +endobj +330 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 412.987950 99.933071 394.237950 ] +/BS << +/W 0 +>> +/Dest (cb9-5) +>> +endobj +331 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 394.040928 99.933071 375.290928 ] +/BS << +/W 0 +>> +/Dest (cb9-6) +>> +endobj +332 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 375.093907 99.933071 356.343907 ] +/BS << +/W 0 +>> +/Dest (cb9-7) +>> +endobj +333 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 356.146885 99.933071 337.396885 ] +/BS << +/W 0 +>> +/Dest (cb9-8) +>> +endobj +334 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 337.199864 99.933071 318.449864 ] +/BS << +/W 0 +>> +/Dest (cb9-9) +>> +endobj +335 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 318.252842 99.933071 299.502842 ] +/BS << +/W 0 +>> +/Dest (cb9-10) +>> +endobj +336 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 299.305821 99.933071 280.555821 ] +/BS << +/W 0 +>> +/Dest (cb9-11) +>> +endobj +337 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 280.358799 99.933071 261.608799 ] +/BS << +/W 0 +>> +/Dest (cb9-12) +>> +endobj +338 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 261.411778 99.933071 242.661778 ] +/BS << +/W 0 +>> +/Dest (cb9-13) +>> +endobj +339 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 242.464756 99.933071 223.714756 ] +/BS << +/W 0 +>> +/Dest (cb9-14) +>> +endobj +340 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 223.517735 99.933071 204.767735 ] +/BS << +/W 0 +>> +/Dest (cb9-15) +>> +endobj +341 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 204.570714 99.933071 185.820714 ] +/BS << +/W 0 +>> +/Dest (cb9-16) +>> +endobj +342 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 185.623692 99.933071 166.873692 ] +/BS << +/W 0 +>> +/Dest (cb9-17) +>> +endobj +343 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 166.676671 99.933071 147.926671 ] +/BS << +/W 0 +>> +/Dest (cb9-18) +>> +endobj +344 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 147.729649 99.933071 128.979649 ] +/BS << +/W 0 +>> +/Dest (cb9-19) +>> +endobj +345 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 128.782628 99.933071 110.032628 ] +/BS << +/W 0 +>> +/Dest (cb9-20) +>> +endobj +346 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 109.835606 99.933071 91.085606 ] +/BS << +/W 0 +>> +/Dest (cb9-21) +>> +endobj +347 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 90.888585 99.933071 72.138585 ] +/BS << +/W 0 +>> +/Dest (cb9-22) +>> +endobj +348 0 obj +<< +/Filter /FlateDecode +/Length 4926 +>> +stream +x][$m~_ `p8 yGuR]xv0;]D$#R29sI/~=Y_?7YJ]ϿHAH&)NrvoN_qgoJ_DeT@.($f稐3R4D1dH +)C@Ʌ-@ i$ +',pZA,Q +ͱѼ31_TZ%B01cъ1rhYX# PIEb* FKctBːڕUd:VᒉTZ攷4opj4 C7JEabRhDEAkY? 8!X68ƐD* `55% |J' |2e |.R + †bI@N`! SԐ 8B\YB! ,aASа m ڢaAܒ-NÂ%뢆 1r$  T4,԰ taI (G p7p7Ti +a &  3`V 0jX]1>gaw}aw} Pi[<7p׳pד@Q Ǫpz8Y߃惆`f<  +ܟb@+r0peLJ+%beb,0p*hX]gְoQ uZw/V"n` !а8 Uiؒ ,y @ iX06q ^4,]!hX4װq h0[Ŧ kfXwMEÂaALRೆ +L5,łOEkְ Y0 KAJakSB$Y$Y2hX[M E R԰Hfr!Ұ qD>Qhћ*-撆)+v 2FaHa͑@pE&{DA 9 ,5,>#ˌVxhjFV̬aAh$T԰fLH `B̏$۬a$HsTd L% Y@7%L YiXPl4&kXZ 7=iX|3kXPEOIÂB4Ȃ(v LB thB% @^]Ȓo&uȤрL@V-3%ꊁ2|d`VI]3_ɑE2GunDge EKT3s,0 BtsQ-S>~9%f1_0݀+S2D16"D*CHX\DHbrZdH,2;3hTzw':,w A7|8vm\yAdeM12YBUޕ.ȿÎr }$H/Le6ҙXv<b$$"+ hl_NrK%ϟ~9OOn?wr'/ڿ3c/e~Y -/^^D/?ߗі !erWle Q3-["[Z۰#cϊHb-3Y͗A9|?K@!EڽA19ăJcLLap "!:Km:RwN4Yo;5=,S9/6= [J9{Dr;zC=ɶA[Ln32l^y$dL::eNZkIPiP=bd'(YbSGMgZv˳N#x$5pE*n*HŷSaT8G[ݥU$UifIAW0;l[N(01y cs|]CZ++>Dy]Y=N%_HPr V|JLjΨ{ 'ֆ]mg|X=vv i-G}ivӮ ,E3nWxZ"$L)(Ÿ#6o%{ĩg@]5X U~82ȓw>YoR>2W#yFI8X{(h rl,b>bcSFJU a͕iI1$zܽ'kk59*{-gd_giMA.k6aTڗڔK=2S6Q4e6䲎|_gABĥQj +p9<ՎI\\"]u<`0ǘwՍ5,%my޳폨^FMrg} ֒CM>ț=wQ5YIxz9V'^ 썋3ǚ| G@tͶ=tu> 4˖8$()Ikv+@Ft!K &nGX3I^'M+HIw$ZJG$#w?-#DKDK񴒶r(4-{B㐳lcLp04>ÀYyʻ#YNhyOhИ\0ǚ|%g$+>Kц|8b;L#)l:Q#gs+qu> )݁Gn +`3l}ݾ};tuv= STZ}D=޿kC`/5&~گ{!X}nV}H[ rƍVM};l25yO99HB:V Gzҫ:g}` xcyݐE'HZkf[粫c) v7t:YqM|wv_,]і6}S{w#U=QuywKVޟ+Ns5zUxuwTC}b:&Iż_y1-z_yt*i\siձw0L[!#Yn6wEa)[V(w}ϲ[A5t]-H= +/q6ҺQikn0i]]Y-W9T8'w-Se >[|Knη: g[O E?VyW:_Xt]|Θ~>^nlU;S%ȸx…FOh桼>3l;̩t5q}mftpBV3QhV@kCo-q8)y`zsalRyA \6KyG]lq5:AcGFQur)ueJ!Si HuXuX Oʹ6!(o!'wSe;nRw|Gq6>Nvη—XT?NFo[9Fw뫜(OK#wV]ik%sOC!{t@(UN-<VnPJP┺]<=IA,_{-j +endstream +endobj +349 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 348 0 R +/Resources 4 0 R +/Annots [ 350 0 R 351 0 R 352 0 R 353 0 R 354 0 R 355 0 R 356 0 R 357 0 R 358 0 R 359 0 R 360 0 R 361 0 R 362 0 R 363 0 R 364 0 R 365 0 R 366 0 R 367 0 R 368 0 R 369 0 R 370 0 R 371 0 R 372 0 R ] +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +350 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 799.173057 99.933071 780.423057 ] +/BS << +/W 0 +>> +/Dest (cb9-23) +>> +endobj +351 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 780.226036 99.933071 761.476036 ] +/BS << +/W 0 +>> +/Dest (cb9-24) +>> +endobj +352 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 761.279014 99.933071 742.529014 ] +/BS << +/W 0 +>> +/Dest (cb9-25) +>> +endobj +353 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 742.331993 99.933071 723.581993 ] +/BS << +/W 0 +>> +/Dest (cb9-26) +>> +endobj +354 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 723.384971 99.933071 704.634971 ] +/BS << +/W 0 +>> +/Dest (cb9-27) +>> +endobj +355 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 704.437950 99.933071 685.687950 ] +/BS << +/W 0 +>> +/Dest (cb9-28) +>> +endobj +356 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 685.490928 99.933071 666.740928 ] +/BS << +/W 0 +>> +/Dest (cb9-29) +>> +endobj +357 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 666.543907 99.933071 647.793907 ] +/BS << +/W 0 +>> +/Dest (cb9-30) +>> +endobj +358 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 647.596885 99.933071 628.846885 ] +/BS << +/W 0 +>> +/Dest (cb9-31) +>> +endobj +359 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 628.649864 99.933071 609.899864 ] +/BS << +/W 0 +>> +/Dest (cb9-32) +>> +endobj +360 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 609.702842 99.933071 590.952842 ] +/BS << +/W 0 +>> +/Dest (cb9-33) +>> +endobj +361 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 590.755821 99.933071 572.005821 ] +/BS << +/W 0 +>> +/Dest (cb9-34) +>> +endobj +362 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 571.808799 99.933071 553.058799 ] +/BS << +/W 0 +>> +/Dest (cb9-35) +>> +endobj +363 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 552.861778 99.933071 534.111778 ] +/BS << +/W 0 +>> +/Dest (cb9-36) +>> +endobj +364 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 533.914756 99.933071 515.164756 ] +/BS << +/W 0 +>> +/Dest (cb9-37) +>> +endobj +365 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 514.967735 99.933071 496.217735 ] +/BS << +/W 0 +>> +/Dest (cb9-38) +>> +endobj +366 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 496.020714 99.933071 477.270714 ] +/BS << +/W 0 +>> +/Dest (cb9-39) +>> +endobj +367 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 477.073692 99.933071 458.323692 ] +/BS << +/W 0 +>> +/Dest (cb9-40) +>> +endobj +368 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 458.126671 99.933071 439.376671 ] +/BS << +/W 0 +>> +/Dest (cb9-41) +>> +endobj +369 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 439.179649 99.933071 420.429649 ] +/BS << +/W 0 +>> +/Dest (cb9-42) +>> +endobj +370 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 420.232628 99.933071 401.482628 ] +/BS << +/W 0 +>> +/Dest (cb9-43) +>> +endobj +371 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 401.285606 99.933071 382.535606 ] +/BS << +/W 0 +>> +/Dest (cb9-44) +>> +endobj +372 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 340.338585 101.433071 321.588585 ] +/BS << +/W 0 +>> +/Dest (cb10-1) +>> +endobj +373 0 obj +<< +/Filter /FlateDecode +/Length 4729 +>> +stream +x]K$ȳH" c>腱SmcQc`QzD$3#f]Ȫ RE~!Z|+395)X%6g|R:Wbן@ɤ| ~.⯗mw]󋰘L)٠ +X֑ +CTXb) bĞoY bpAcA >+ 1[V>9XÕykU &BQygb.K^1b %8f SN/BdUd9KC;Ee*h+dB*gsʷ%ų>YCH_Xdxi@@, Uy +kX@D3iX@@[hX@[iX@[.jX@\i5, . T4, `.;5, .=iX@@] +kX@]5, .0p5UdkXt42iX]r6iX]װj`Q؊9[ e BK>H0p܂>Xaw=a  w!AO V{Ne}||M])k?OX|xLm B*"$|W%#!0n"&1n/n+4dҙzoѢKJ!Y7uR"1`J~|_?槯+k rV/ > =`v0s?-6eۏ46R]k?,]5׵+=v6D<ֺBCڇ_YoD[>m沿Кk3qǵUXnk׭&4KkZ燵Q-M:T/VjM8hik?R|ꊿ,zj ӽƥ*eo5޶=}a_h1ѼʢbnPڷ'E]$DD:.p!$@TLm}!uޅ0+`d07CK E ch?>fj{:Koaa+j*_aRw.ia~F|EF&1uϭERl !Wʿ joXV^su( ײm0x̋SWܠ=㢿@^Q"x(䭗޾_[fr+ Ĝ5XdM񰜩<=*}P@@v(NsUG1bZg Y.Sj&{e, +Ng'<:1sXJlQgH:o;ewzuNvamW D +Bsew4!xyBɘX};G6mj`Vva$cL-M8ڃ^e{hf#~L-GltyOMjpc@*Y|x2\ܶ⾜vN6d٦':!}76]h9Yi t}gqθ:ڈd3"Ͷ9~ju>nbaP=V*OpÇC8Qw{𫪧r(z\2BF#&=,yֶ "5#D?YrBxߚ֖$e&[byc3Wkjy4fF#CSw&32ޒ9wet:Oɲ pc?$ +~5"f!bHu]3ô{k.v1rctvM)؟0Xy Nہ9*In۽hR|dUs qY^lanc6G_G +}S_U .Iٶ -mKm}?q|?en[76no{Z'y6=r7%3\=l @M΅цUʳf=^J~d$Y8_E2[ٷA~uCׯ7; P8/)+׾[bGЪɩl[!ꗎ"jFwKƬdI Wb eG!ّhsk%]uq4Op>"\-*UkM5&JVqRH/mM3Q[bnĽ\iE-H^*i~2LrWB:ZI4|R%}m_.'%$tXaC3HSAM +hr^"t O⸛_3F'⛝ULuXӀ4NBFmv֗V)/Kwm&z*M<[VaM?+gmBAݔ_7!gl,Cowcי;9۝l_j`=YL?NYfS|zgqPܼھ'"</y}$'gY/O0%{wlݐH&'d<݈m& Q64kq yT[IuL CwӕfN=c3rI 1@M*\̷> +endobj +375 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 709.923057 101.433071 691.173057 ] +/BS << +/W 0 +>> +/Dest (cb13-1) +>> +endobj +376 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 690.976036 99.933071 672.226036 ] +/BS << +/W 0 +>> +/Dest (cb13-2) +>> +endobj +377 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 672.029014 99.933071 653.279014 ] +/BS << +/W 0 +>> +/Dest (cb13-3) +>> +endobj +378 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 634.134971 99.933071 615.384971 ] +/BS << +/W 0 +>> +/Dest (cb13-4) +>> +endobj +379 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 596.240928 99.933071 577.490928 ] +/BS << +/W 0 +>> +/Dest (cb13-5) +>> +endobj +380 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 577.293907 99.933071 558.543907 ] +/BS << +/W 0 +>> +/Dest (cb13-6) +>> +endobj +381 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 558.346885 99.933071 539.596885 ] +/BS << +/W 0 +>> +/Dest (cb13-7) +>> +endobj +382 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 539.399864 99.933071 520.649864 ] +/BS << +/W 0 +>> +/Dest (cb13-8) +>> +endobj +383 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 520.452842 99.933071 501.702842 ] +/BS << +/W 0 +>> +/Dest (cb13-9) +>> +endobj +384 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 501.505821 99.933071 482.755821 ] +/BS << +/W 0 +>> +/Dest (cb13-10) +>> +endobj +385 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 482.558799 99.933071 463.808799 ] +/BS << +/W 0 +>> +/Dest (cb13-11) +>> +endobj +386 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 463.611778 99.933071 444.861778 ] +/BS << +/W 0 +>> +/Dest (cb13-12) +>> +endobj +387 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 444.664756 99.933071 425.914756 ] +/BS << +/W 0 +>> +/Dest (cb13-13) +>> +endobj +388 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 425.717735 99.933071 406.967735 ] +/BS << +/W 0 +>> +/Dest (cb13-14) +>> +endobj +389 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 406.770714 99.933071 388.020714 ] +/BS << +/W 0 +>> +/Dest (cb13-15) +>> +endobj +390 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 387.823692 99.933071 369.073692 ] +/BS << +/W 0 +>> +/Dest (cb13-16) +>> +endobj +391 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 368.876671 99.933071 350.126671 ] +/BS << +/W 0 +>> +/Dest (cb13-17) +>> +endobj +392 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 349.929649 99.933071 331.179649 ] +/BS << +/W 0 +>> +/Dest (cb13-18) +>> +endobj +393 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 330.982628 99.933071 312.232628 ] +/BS << +/W 0 +>> +/Dest (cb13-19) +>> +endobj +394 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 312.035606 99.933071 293.285606 ] +/BS << +/W 0 +>> +/Dest (cb13-20) +>> +endobj +395 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 293.088585 99.933071 274.338585 ] +/BS << +/W 0 +>> +/Dest (cb13-21) +>> +endobj +396 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 274.141563 99.933071 255.391563 ] +/BS << +/W 0 +>> +/Dest (cb13-22) +>> +endobj +397 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 255.194542 99.933071 236.444542 ] +/BS << +/W 0 +>> +/Dest (cb13-23) +>> +endobj +398 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 236.247520 99.933071 217.497520 ] +/BS << +/W 0 +>> +/Dest (cb13-24) +>> +endobj +399 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 217.300499 99.933071 198.550499 ] +/BS << +/W 0 +>> +/Dest (cb13-25) +>> +endobj +400 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 198.353477 99.933071 179.603477 ] +/BS << +/W 0 +>> +/Dest (cb13-26) +>> +endobj +401 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 179.406456 99.933071 160.656456 ] +/BS << +/W 0 +>> +/Dest (cb13-27) +>> +endobj +402 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 160.459434 99.933071 141.709434 ] +/BS << +/W 0 +>> +/Dest (cb13-28) +>> +endobj +403 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 141.512413 99.933071 122.762413 ] +/BS << +/W 0 +>> +/Dest (cb13-29) +>> +endobj +404 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 122.565391 99.933071 103.815391 ] +/BS << +/W 0 +>> +/Dest (cb13-30) +>> +endobj +405 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 103.618370 99.933071 84.868370 ] +/BS << +/W 0 +>> +/Dest (cb13-31) +>> +endobj +406 0 obj +<< +/Filter /FlateDecode +/Length 4922 +>> +stream +x]]#muH"8)r`zyo=Nzѿ߇^.!GF#L%E>|dM +jEer2g!P2.??:M)ѻp7a1SAa#Iɇ3R4=Y bpAcA >+ 1=@ k,JH +Yἵ*Xf +31_T^%LLxCx YBNx)'! E*H%⡝2dueYlN2dfW9=`B&9`Q,ц K 5A_rVՋw%d>ʃ,| |,6E AFpIYp次pJ/p,I<4 YG AH0p܂6sw}^zQX5B@a3a +?ŧV5,]1|*qoeb,.&*Ұ wY`5,]G5pb5,] Aq4,f骼blYÂ,y 0 $*/Ӱ`l,P "p{Ѱ wm Ұ w-#Qyװq xp[ŧkFXwMEÂܕiSÂೆ +̦5,łOEaA @ʃ,X֦P$Y{ ,4,@D`C Y)jX$30kXBD>jMNksIÂ`*v 2NaHaϱ@L5,5,@ sdY jX}FKBYÂHa&RAIH~l"QykXt3kX3dkX۔|3iXP,dAħaAHa`)vY_O=8JbE|pLmEBH".,.n"$bLܾȐXdvf/2zw'uAyTR#TNy^~4Y,Yi AY&DuLRI 9яDXx3%,o + tYn؈ӯǟ~O?Z7?8˵wULBY# +\TkT!7U7H8̀Mk5cҭew{{Ё$=l8\cj R҄oD$]2do]JZuTȰJ|mk@ve~޴*/ckȿ"t{"\!D~grwc t#!5#$6zu誛nmljSw7v{% +-z "S}mIKlFm2ՖG쐦"|Zms@|yanySUjs1>?bVt5>+9ɢ,8\8СyFgWСj}ؓ{|?>B+OJ? 9kD}r N~iR0y2s]ݷJH2wQ["١b"5ٛHIQ} @irao}e|}W'⣰w :l? Euu޶=}g&vto$ |CIuqܵ{b}km۲ܥLpl1V\L8^$;GAa ꃫ~1WeBJ1!n~J$m-mkt]kcw S`=IR{2g_f7hc~<7!lV{$Oq&w֛DSXףAeRIS&d](.i3a#0zYY;204b/cJNyAߑ!<.=-ÛESR32O:`TjϬQ(?&S"֞YlvQzݐr[ݷpz{FmPVeY&y(BПK_`g~b6fv-NDW-6SjtVsz  z:K̜V{P|vu^\a +l #//y.A,lU[ "DI@ᅈ'Ȑa7 =6$s=IR;, &oX:Bh +Q8"Mo ^#6?I +{.{="n7."9<ʑ檼ȫM"6l=~cv^SU^GA薥ϧ( Nbr}꼎][g wJCG$!#wCp!#\w.{74Y)玵dX}*GH0,>0*CU +EX" Bf *yKd펃|*ehhz(cAYE@H;emhVAV'V1Go|>ûB;;Z\ :f;2"bq>ӟ:"ykX$ns2}dJ0Ѭ窼yA*9L#4`=nw6:Q[h&m8ӏaVKSU^F:-K+1D`GYdQ[ѵpGe.nv=}?@a=A};1J3Fs|F +y>jsw(ܰcn}H[/׃`REU%БSU!s!#sHT!㱐rmBR;Lc; +'3ĹzŹIv;nߥmt,ťxHd0)^ᐹFz/S_UxR&ee.rdJ1%Es uX-Ӄ;?->ޖ]W m[_CXV[÷}.OE[A*uиwN}Œ];+UYyu d2 bKE6Qu:L=`\JG87W.]\&GGIǹ)ۻ0bJt%|q ,-fHe{j'|TޕvZ` +rf^  Hgwl($-I)S{, +bϗ:]sָCJ}mwzUw\VT/6qigiMO_az[|~Xmzazjci䅻T.eK܎=rUIzt/ neEx'd]_듣7Ӷ_/:$'Bf6~,mmZ L.._Kk\Wֻa&'~JlұBDd{$ YNoφD qY e{!Yİ'ڡNeֲ}-mAlse}nR赩UHW BF k6 I!vm1?y!Ր+k9vR-6RʽjY5EN)^U-$?G1eֱy#d[͎q +I):VذHFζK"\T\irn{ШMJ)\CE\rt6bA4Ջ<C!yܧٱVb(Ƌavce#"㾈WjFƄOR6P+Wjm{Ԃ=h5jK0v+m:Q4ͮbZeKU-bNۨ0JSbG%Jע%jP9d%BcF^Vھoζ6 {dj{+=;:WZo] q]]"Ol3q4rwk%ud a&'+F9f M9We-ݧe)ÜݭŸeA]&IwH,?S +mO r +endstream +endobj +407 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 406 0 R +/Resources 4 0 R +/Annots [ 408 0 R 409 0 R 410 0 R 411 0 R 412 0 R 413 0 R 414 0 R 415 0 R 416 0 R 417 0 R 418 0 R 419 0 R 420 0 R 421 0 R 422 0 R 423 0 R 424 0 R 425 0 R 426 0 R 427 0 R 428 0 R 429 0 R 430 0 R 431 0 R 432 0 R 433 0 R 434 0 R 435 0 R 436 0 R 437 0 R 438 0 R ] +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +408 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 799.173057 99.933071 780.423057 ] +/BS << +/W 0 +>> +/Dest (cb13-32) +>> +endobj +409 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 780.226036 99.933071 761.476036 ] +/BS << +/W 0 +>> +/Dest (cb13-33) +>> +endobj +410 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 761.279014 99.933071 742.529014 ] +/BS << +/W 0 +>> +/Dest (cb13-34) +>> +endobj +411 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 742.331993 99.933071 723.581993 ] +/BS << +/W 0 +>> +/Dest (cb13-35) +>> +endobj +412 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 723.384971 99.933071 704.634971 ] +/BS << +/W 0 +>> +/Dest (cb13-36) +>> +endobj +413 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 704.437950 99.933071 685.687950 ] +/BS << +/W 0 +>> +/Dest (cb13-37) +>> +endobj +414 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 685.490928 99.933071 666.740928 ] +/BS << +/W 0 +>> +/Dest (cb13-38) +>> +endobj +415 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 666.543907 99.933071 647.793907 ] +/BS << +/W 0 +>> +/Dest (cb13-39) +>> +endobj +416 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 647.596885 99.933071 628.846885 ] +/BS << +/W 0 +>> +/Dest (cb13-40) +>> +endobj +417 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 628.649864 99.933071 609.899864 ] +/BS << +/W 0 +>> +/Dest (cb13-41) +>> +endobj +418 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 609.702842 99.933071 590.952842 ] +/BS << +/W 0 +>> +/Dest (cb13-42) +>> +endobj +419 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 590.755821 99.933071 572.005821 ] +/BS << +/W 0 +>> +/Dest (cb13-43) +>> +endobj +420 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 571.808799 99.933071 553.058799 ] +/BS << +/W 0 +>> +/Dest (cb13-44) +>> +endobj +421 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 552.861778 99.933071 534.111778 ] +/BS << +/W 0 +>> +/Dest (cb13-45) +>> +endobj +422 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 533.914756 99.933071 515.164756 ] +/BS << +/W 0 +>> +/Dest (cb13-46) +>> +endobj +423 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 514.967735 99.933071 496.217735 ] +/BS << +/W 0 +>> +/Dest (cb13-47) +>> +endobj +424 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 496.020714 99.933071 477.270714 ] +/BS << +/W 0 +>> +/Dest (cb13-48) +>> +endobj +425 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 477.073692 99.933071 458.323692 ] +/BS << +/W 0 +>> +/Dest (cb13-49) +>> +endobj +426 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 458.126671 99.933071 439.376671 ] +/BS << +/W 0 +>> +/Dest (cb13-50) +>> +endobj +427 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 439.179649 99.933071 420.429649 ] +/BS << +/W 0 +>> +/Dest (cb13-51) +>> +endobj +428 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 420.232628 99.933071 401.482628 ] +/BS << +/W 0 +>> +/Dest (cb13-52) +>> +endobj +429 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 382.338585 99.933071 363.588585 ] +/BS << +/W 0 +>> +/Dest (cb13-53) +>> +endobj +430 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 363.391563 99.933071 344.641563 ] +/BS << +/W 0 +>> +/Dest (cb13-54) +>> +endobj +431 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 344.444542 99.933071 325.694542 ] +/BS << +/W 0 +>> +/Dest (cb13-55) +>> +endobj +432 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 325.497520 99.933071 306.747520 ] +/BS << +/W 0 +>> +/Dest (cb13-56) +>> +endobj +433 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 306.550499 99.933071 287.800499 ] +/BS << +/W 0 +>> +/Dest (cb13-57) +>> +endobj +434 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 287.603477 99.933071 268.853477 ] +/BS << +/W 0 +>> +/Dest (cb13-58) +>> +endobj +435 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 268.656456 99.933071 249.906456 ] +/BS << +/W 0 +>> +/Dest (cb13-59) +>> +endobj +436 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 249.709434 99.933071 230.959434 ] +/BS << +/W 0 +>> +/Dest (cb13-60) +>> +endobj +437 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 230.762413 99.933071 212.012413 ] +/BS << +/W 0 +>> +/Dest (cb13-61) +>> +endobj +438 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 169.815391 101.433071 151.065391 ] +/BS << +/W 0 +>> +/Dest (cb14-1) +>> +endobj +439 0 obj +<< +/Filter /FlateDecode +/Length 4511 +>> +stream +x\K$ s+HÀ $`` 0rXwv*=];ۋLzPRIp_fgr.)駓5)XWKl*[pv ? +I%/'o897o븳7Db2)d*`YG ++QaEg0R4=_Ƃ}VXb'G׬1|r +!),g!֪<b+<Lң\|Qy <#21 Nc Y1d : KpVA4^,ɪ B-ȯ`oSJ^SV:Me&p'H +Av02>*BDlP #7\<,M< JNH\ʹtxhIJZj#DQjf1ـ+qtU,VH"P,$,o +dZX6Gu4>x=xK}oyo)\eǕNL?r?hw<݀"x\Gϓ8,`="' U4Ɗ%8ڕ>?nLLTD~})9jS@^`n@.Pp?ty7FlYVjݱ*cwɦ{;Lgll;yGb~}˳̹o6d ,vov<_ײ4f{al|ֺ؇L{,6墩}8ߕvu#꿛{[̞$[uhB1 IzuT<4V=r坳ln#@\^ڹHhvym+ޕ[r1:m|utRd$ѲEiZPX9Yc>vҷ(S㞑ɫXl[=ڼ[r qbʗDzw=u[nJwkԥТ$~mǮtq^|?Z ,?O(h+ȴ"<$c(d!{6Hmr,>?&JlcM-MQz$XB1<D@|R.FW]uNv:VYVY#0I~Dڐ(GY:ȇX}GVx?Z"Z oyx64lrɄGڿ[3%qR07L|/ٱ:o|l(cm(laIY N/K-=6⩞U[Mm,3S\mG[Z_Dx̳߭K{G֝.o8S*= AX.>2CU~mDBmiYIesr 4f/[wx(28yw=YFi{+Ѻݻv!#ӽ-/x6uzٽܻǷC(^Wl߷CU0YbiCvSU/9EI"crQ"x(57'Rz>~oZl-A'&[É9s#;rLJa|cRtVA' (F c:cx2wR[[i?<ꔣPeg:Ov{=ewz +v;%ڕ _0~LR}۽g/_@ "f2$ťu4rc+aˠlIX̳nO5bv&z4`;Ty& "I.}FT?'D߃uF5דE/vءpjW//[= +Gͫˍ'oq|#xig@wryU!<3s4ucUG1r+ a,8D.'f)3{V=##<-QNp5i+elɔ/Vo2 +Ǿ9ȶ#ɎyafMrX"׋$wt߮W^R8˻eWnt>,@mNshRuuJaihסE[NFy@\"FWX;FϼWy/78[nogN'y60˲j~:zի^4X62㩜*e{&W7j/K]S|ahzXT'ǓNv1e'/ljd!~|.'T%)9N8nh~h"VUc]eָ,0Ob eGC#;5N &K]uq4Ot#^N*7ZSMvMBN d.'mM3&Y 17C^r +rR˭>Sʣji S.M$_|RsKC}(v㯈Sr +I)VLo&n'g;ePSnk6\/P:U!\k`~ovVI3=vaM(;˥mv֗V)/Kwm&z*M<[VaM?+gmBAݔ_6!gl,CowcΗ;N6߈xOϭAk[|zgqP/ܼ|9L?O5y +^rHN>G3gY_#~u%{wlݐH&/1Fn3=O?.YCr[+hR`o%]GP׹31 NdMWn ̈́;N=ܹ{y68tCrsIy8x̃G#$&$EGHTsx:,Cbs']"?SA0H#M7ZT)0sNFEOvϞU<@;e N=Y h=IpZaщG@;FtE lS)՚ZꈳsA0JWbg%^v[((sɬ\SyF}h{C82S;;KݮV;!./ P˖Xc|+qF^]^H+{t&|u^(SSSzb!NVЀ9{TXo d(o%Iz.Y u eUU +endstream +endobj +440 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 439 0 R +/Resources 4 0 R +/Annots [ 441 0 R 442 0 R 443 0 R 444 0 R 445 0 R 446 0 R 447 0 R 448 0 R 449 0 R 450 0 R 451 0 R 452 0 R 453 0 R 454 0 R 455 0 R 456 0 R 457 0 R 458 0 R 459 0 R 460 0 R 461 0 R 462 0 R ] +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +441 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 541.623057 101.433071 522.873057 ] +/BS << +/W 0 +>> +/Dest (cb17-1) +>> +endobj +442 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 522.676036 99.933071 503.926036 ] +/BS << +/W 0 +>> +/Dest (cb17-2) +>> +endobj +443 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 503.729014 99.933071 484.979014 ] +/BS << +/W 0 +>> +/Dest (cb17-3) +>> +endobj +444 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 465.834971 99.933071 447.084971 ] +/BS << +/W 0 +>> +/Dest (cb17-4) +>> +endobj +445 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 427.940928 99.933071 409.190928 ] +/BS << +/W 0 +>> +/Dest (cb17-5) +>> +endobj +446 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 408.993907 99.933071 390.243907 ] +/BS << +/W 0 +>> +/Dest (cb17-6) +>> +endobj +447 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 390.046885 99.933071 371.296885 ] +/BS << +/W 0 +>> +/Dest (cb17-7) +>> +endobj +448 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 371.099864 99.933071 352.349864 ] +/BS << +/W 0 +>> +/Dest (cb17-8) +>> +endobj +449 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 352.152842 99.933071 333.402842 ] +/BS << +/W 0 +>> +/Dest (cb17-9) +>> +endobj +450 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 333.205821 99.933071 314.455821 ] +/BS << +/W 0 +>> +/Dest (cb17-10) +>> +endobj +451 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 314.258799 99.933071 295.508799 ] +/BS << +/W 0 +>> +/Dest (cb17-11) +>> +endobj +452 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 295.311778 99.933071 276.561778 ] +/BS << +/W 0 +>> +/Dest (cb17-12) +>> +endobj +453 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 276.364756 99.933071 257.614756 ] +/BS << +/W 0 +>> +/Dest (cb17-13) +>> +endobj +454 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 257.417735 99.933071 238.667735 ] +/BS << +/W 0 +>> +/Dest (cb17-14) +>> +endobj +455 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 238.470714 99.933071 219.720714 ] +/BS << +/W 0 +>> +/Dest (cb17-15) +>> +endobj +456 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 219.523692 99.933071 200.773692 ] +/BS << +/W 0 +>> +/Dest (cb17-16) +>> +endobj +457 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 200.576671 99.933071 181.826671 ] +/BS << +/W 0 +>> +/Dest (cb17-17) +>> +endobj +458 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 181.629649 99.933071 162.879649 ] +/BS << +/W 0 +>> +/Dest (cb17-18) +>> +endobj +459 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 162.682628 99.933071 143.932628 ] +/BS << +/W 0 +>> +/Dest (cb17-19) +>> +endobj +460 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 143.735606 99.933071 124.985606 ] +/BS << +/W 0 +>> +/Dest (cb17-20) +>> +endobj +461 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 124.788585 99.933071 106.038585 ] +/BS << +/W 0 +>> +/Dest (cb17-21) +>> +endobj +462 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 105.841563 99.933071 87.091563 ] +/BS << +/W 0 +>> +/Dest (cb17-22) +>> +endobj +463 0 obj +<< +/Filter /FlateDecode +/Length 4905 +>> +stream +x]]$}_qXDQa xF7ugauR "E%GJv/q+gr.)_/]IlTbu1OgC2J(Tاv_7}˟wZƽxSJ_S +X֑b6)y ++:[,EcA N-+AP2k, ++C +іUO.h, +',pZElcQygb.K^c1rxpN1tax%PN/BdUt9Kţw֕Ut:Wp!)oy޲Ld5saQ.rQyЅzG%N0odi<.ާ _CMQx\0dBAɄ>R" |abIuyY!jH@xʃ.T +hX@]| h`-64, m.jX@\i5, .4, @.LhX@\vjX@]{ҰkX@] +9jX@] +jXDTyaѐˤaAIÂ܅y 0v5,հ w}A SK>Hg܂6sw=[aAࣆ0j4@a*3a uӟ@+r0pe +%eb,0piX]gְo0z 3:_ ECa7,LWcK0p5,,lld`HNC5,]LEÂܵLaAڀGA^5mFjRh7wֆqo";uR=aHkcd|*!S2p9Y[_ +83>ݵ?|ZP1-L Fn4eN $T)NE }DYbUu̝bxm5P~!}`pxY jAx8YŐVBŪ(6!MwxDX#d5{wtx,\2w=Y"y7_ݱjEe'0X0}P@/d.@p]Hp+$6HGn-@-d`*i:E-:Szo$0³m G"#"+ܐ,y;-x1dSg1C&uu?&DH>-f{ou{?Vc5-Gٶ*k_ߏ|N&bEn"?&4'[+|'lUL/rlA8n w㚐-EװlEz+n/}3(|qʹ5#`ԗdeG|d~bV.und%zĹrّGZurHr1VӒ dξx&#r]_ae<4n2O-^׫>mg$^7t4J3G-Fմ:Rq[[nm^aw-38[߶3vtQ=R du)7mc2OzD+{ا ̖Q9;:y$ISYw CCܠɗq9v{h#~,i3SeOy{1X"ګ6e`A%jEau?F>*z@n<| vPHqwoΠEQ@<\Ww~mk܇t_>zH!`sq+=^G\t B>~MO/ ;FT?^ץ\Z6v0z] +vD .|VNs9~F{9#YaK伽56G:um4s\ CaFO`zuߦCW04B{2O~G($s]cmGLEY㘣)D;El$oO%DgXB˽`BuOR}rxG朐uuP/%2,u9#ݼg``gիvv?mݗ3Tz"}Ez~,ȭ{޽9 ݢ0u+E93BwRfc͟cYN zgMw}Gc&Kç=#I?_6lL %B9wͨ%239VYۿux-Ďl~S=sݜ>&9h+U?mwLupwCe~(W7m4JNrEc` >:ֽ#9Jo a4\i.i*+k +mr@ؙ,? WTb0e[Mp,5"okP_Ks}׹n>uyzAtK}SKUKb&[h)U*u@B'zQإjMQCļRRN!^rsm\G*R*)׎WhS$|eF6oI) +,/mҨ)/]\hrD} +TA դv߬u'XQ'h=BjeB?.g>_qo+Z+r?6ABn}`]ney:{uhRUuaGtgk ѱiC3$/Zᢿ\ƣ6%tUJ);A\k4VjAԮC?< m^h]tESW\hY(}ڽ>> +endobj +465 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 799.173057 99.933071 780.423057 ] +/BS << +/W 0 +>> +/Dest (cb17-23) +>> +endobj +466 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 780.226036 99.933071 761.476036 ] +/BS << +/W 0 +>> +/Dest (cb17-24) +>> +endobj +467 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 761.279014 99.933071 742.529014 ] +/BS << +/W 0 +>> +/Dest (cb17-25) +>> +endobj +468 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 742.331993 99.933071 723.581993 ] +/BS << +/W 0 +>> +/Dest (cb17-26) +>> +endobj +469 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 723.384971 99.933071 704.634971 ] +/BS << +/W 0 +>> +/Dest (cb17-27) +>> +endobj +470 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 704.437950 99.933071 685.687950 ] +/BS << +/W 0 +>> +/Dest (cb17-28) +>> +endobj +471 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 685.490928 99.933071 666.740928 ] +/BS << +/W 0 +>> +/Dest (cb17-29) +>> +endobj +472 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 666.543907 99.933071 647.793907 ] +/BS << +/W 0 +>> +/Dest (cb17-30) +>> +endobj +473 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 647.596885 99.933071 628.846885 ] +/BS << +/W 0 +>> +/Dest (cb17-31) +>> +endobj +474 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 628.649864 99.933071 609.899864 ] +/BS << +/W 0 +>> +/Dest (cb17-32) +>> +endobj +475 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 609.702842 99.933071 590.952842 ] +/BS << +/W 0 +>> +/Dest (cb17-33) +>> +endobj +476 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 590.755821 99.933071 572.005821 ] +/BS << +/W 0 +>> +/Dest (cb17-34) +>> +endobj +477 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 571.808799 99.933071 553.058799 ] +/BS << +/W 0 +>> +/Dest (cb17-35) +>> +endobj +478 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 552.861778 99.933071 534.111778 ] +/BS << +/W 0 +>> +/Dest (cb17-36) +>> +endobj +479 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 533.914756 99.933071 515.164756 ] +/BS << +/W 0 +>> +/Dest (cb17-37) +>> +endobj +480 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 514.967735 99.933071 496.217735 ] +/BS << +/W 0 +>> +/Dest (cb17-38) +>> +endobj +481 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 496.020714 99.933071 477.270714 ] +/BS << +/W 0 +>> +/Dest (cb17-39) +>> +endobj +482 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 477.073692 99.933071 458.323692 ] +/BS << +/W 0 +>> +/Dest (cb17-40) +>> +endobj +483 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 458.126671 99.933071 439.376671 ] +/BS << +/W 0 +>> +/Dest (cb17-41) +>> +endobj +484 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 439.179649 99.933071 420.429649 ] +/BS << +/W 0 +>> +/Dest (cb17-42) +>> +endobj +485 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 420.232628 99.933071 401.482628 ] +/BS << +/W 0 +>> +/Dest (cb17-43) +>> +endobj +486 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 401.285606 99.933071 382.535606 ] +/BS << +/W 0 +>> +/Dest (cb17-44) +>> +endobj +487 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 382.338585 99.933071 363.588585 ] +/BS << +/W 0 +>> +/Dest (cb17-45) +>> +endobj +488 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 363.391563 99.933071 344.641563 ] +/BS << +/W 0 +>> +/Dest (cb17-46) +>> +endobj +489 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 344.444542 99.933071 325.694542 ] +/BS << +/W 0 +>> +/Dest (cb17-47) +>> +endobj +490 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 325.497520 99.933071 306.747520 ] +/BS << +/W 0 +>> +/Dest (cb17-48) +>> +endobj +491 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 306.550499 99.933071 287.800499 ] +/BS << +/W 0 +>> +/Dest (cb17-49) +>> +endobj +492 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 287.603477 99.933071 268.853477 ] +/BS << +/W 0 +>> +/Dest (cb17-50) +>> +endobj +493 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 268.656456 99.933071 249.906456 ] +/BS << +/W 0 +>> +/Dest (cb17-51) +>> +endobj +494 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 249.709434 99.933071 230.959434 ] +/BS << +/W 0 +>> +/Dest (cb17-52) +>> +endobj +495 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 211.815391 99.933071 193.065391 ] +/BS << +/W 0 +>> +/Dest (cb17-53) +>> +endobj +496 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 192.868370 99.933071 174.118370 ] +/BS << +/W 0 +>> +/Dest (cb17-54) +>> +endobj +497 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 173.921348 99.933071 155.171348 ] +/BS << +/W 0 +>> +/Dest (cb17-55) +>> +endobj +498 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 154.974327 99.933071 136.224327 ] +/BS << +/W 0 +>> +/Dest (cb17-56) +>> +endobj +499 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 136.027305 99.933071 117.277305 ] +/BS << +/W 0 +>> +/Dest (cb17-57) +>> +endobj +500 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 117.080284 99.933071 98.330284 ] +/BS << +/W 0 +>> +/Dest (cb17-58) +>> +endobj +501 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 98.133262 99.933071 79.383262 ] +/BS << +/W 0 +>> +/Dest (cb17-59) +>> +endobj +502 0 obj +<< +/Filter /FlateDecode +/Length 3167 +>> +stream +x[Y ~_1T& @ (0 Xycfa$Z4YGτU9JBIۈK:3i [RBi)|~8:W]s cRKXR +ĔV[lcAT4=@ m5y,1VU!hag5 ǂRqXL#˃ *q +J/rȵ +x\ ('x-Sx +YR#x $V%YTjxHry%"@4W! ZY&VȀV=3 HDgY{j +&{l@_cz zD9ɆPT~9}˿OoNE5B$6p!dMhS6!'L1J"2@8r3ly|k!I(eu[ CIv%)_Hҩ F̓UC(g,{J}/uߩ#O֏eERkz_Wȧ?)ؿ'(_XH"9Ek6k +Eҏ%,}N%-Z^X#}uO"6*Mf#_;}|@ QY-qIYWے$/l(e%wMi TSжŤlV=jg$lp\Pv?j \%m|+V+mo~V7c^]1Fr}1 +\5Vva)iNGHnwM=hvw[{t`sXѰ}IGFzD/~Fz[66Z]鵿ٸG^'$yr|건xV79=NZfԀ@c[K##pڷ{5V{f>9ù/G5ToOދ1~76?iv8+}cv 25q()w :v.8'>Ifl7Q6Ŝ/'l+jC_k7Ҡ"TG谓C K+9INݑtk}CqjiXQuNm]SRڦZ%FcJfnk;tD$?RqFDYJ_& ơ*f9ܡ" !u=WsAVJukcuq)gW}Yt?@w.*[u x\yx u \ԅv+?\ijYXw·\ yǺweKlzK6ׯ}I4l!N +63aoۈ=2Rb\ɓ˒3=X&b]ˆmJv%9Pɯ)Z.2.etuNbX֋uauv%#'g90A}Xƌ:ϧv/GΑ XKؓQИ^]>Pv HiWD14괱b2cZWvR5Ťj:VJLUX[gQ-(e͑-UHzmm$ʔt7Sbgss_Ir(_P_Ѻ|JODYɺT +"<ўf,K5row.vȽmBXh!VV{ݴGCJ:>پ X9 +endstream +endobj +503 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 502 0 R +/Resources 4 0 R +/Annots [ 504 0 R 505 0 R 506 0 R ] +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +504 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 799.173057 99.933071 780.423057 ] +/BS << +/W 0 +>> +/Dest (cb17-60) +>> +endobj +505 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 780.226036 99.933071 761.476036 ] +/BS << +/W 0 +>> +/Dest (cb17-61) +>> +endobj +506 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 719.279014 101.433071 700.529014 ] +/BS << +/W 0 +>> +/Dest (cb18-1) +>> +endobj +507 0 obj +<< +/Filter /FlateDecode +/Length 4957 +>> +stream +x]ۊ%}_ '́cY$փ6ÈB'( l gQ[UFY#Ɣjht[eŘDlAK `lQ1H4/I`b\6N"K hd?Vi茖*]j0rElI\` lA -Y ,-XЭSYK\`N qL%q (e ,8ׂ;%. ] (qZk] +XuZI\"-`d%.p]Uw18 +III\&)'q)@cJ\ƣ 5^ x8$.P.(lJ\:c%.@aAX"qO`A$.]}.:¸Bw1QY p`!e$.>`=J`Aע.A1Sk!aK\(\f@1  0 2t Ȗ.tZK\ lA4(qA2 ̠$. $\-霢uf 0P(q V`JQ[@h%.M,sDlI\f %bؤf +؂O€@aq蔒 q`\z3I\zK$xi'Xo" +#3Yhř[-l=5i3LA5 +Vⶁ"s+NwU"!okn$/?bo_C4HAm_.pk@.4]ӂ\RXv-vabDy`M}˱KEΪӈ56&0Hb[^^n0iot{G3:+/{1QVVA]sjeںD +¡w5zdF\t..eM2"/(Ŵziq sq9 +ViS!R/>t긐C(j^U2p)R@; _9ȸ߸l;39b>| /o6 9( tҾ͍j(Fw5y0І|M(mRmPv*ܬ^Uֻ󼻭+DzE#ZwFzz]Wɚ${6;ޠ z Υrn+O><_+ ++]nP@xnR.)?h`F~ldh]xDkMQ7L܏PZ 4F9SZTQ~KZYԅm۲rF9ʋq-zd{c53cSxff>fy集!lxumq"x~Tc;`ͦVkf\F?6|fgGjV%N<6qWcTǮ.-5Su LJ2KwD7\fIe$m< EBdlf7ϫbi" ]G{Q +vN-gWjkکٛkj8~_l%xMӞg{˦6+aPnYTX)tD_ANP> \c!*e2=)e;솇$u{,,4rՒv~WX~w`%r>O[4DNrXDu[|5v+PYtm;AzP[ aݟQ%y޲Gp'}cn~GC乾*FXغ,o!~ wOyאV }| +NlrW<@zB=2ʱy yp?Q~K/W::"ͧ$7fO߼N$<.ꌋrc{eP@E:syelzheO&tV:52?^"4Y[i+.{@B[.yoV.{\#M ,QL?n~x>_Wf~KRS<R"U@TvSk=3Zrm;*2GԾ끝ű䍑{;ZolOm!3ƇC92Oy8d]m^\Y{( kϸ{ maѠuu4o~Λ1v \CxQ\mlqq+.ODֲ'9Q_Ocqe2/lڶuƓ;!鹤T%G޹<rٓqXϮfc)oH3[~ޏY?YlGzY͐U2;<'<zA\YYEϲUq|5ri+ybf!mg yWu2bcO#dC~01[׼޾AtsϨw\ky1ڎλqDwE@s2áy?]ţ4;7 ;T{F_īK|sm#+ƶFқĔ.oG<ƴǮh({zz]ݕZ<0T +5xfH;Vҽi^^p?}CLy][ήo=8ӈS[ۢ([{E3ۻ<":w.Yj"[ި:V,3赘Z=/iYhM tdbӠke;‹-yk {򢫬%Al /i^i#~LuQ{pikIR[N9Vec϶ Z (mJyf։dZl-IRl1E{^iNW*-[g-y[zK&Ax&u=un^wmzdz6J*{(?e'D LNGklJ!C}*:lwׄ*j*l X_e%6>[AVon峄eϯlHKE +7Ӣx#mI??"; +endstream +endobj +508 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 507 0 R +/Resources 4 0 R +/Annots [ 509 0 R ] +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +509 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 563.554475 101.433071 544.804475 ] +/BS << +/W 0 +>> +/Dest (cb21-1) +>> +endobj +510 0 obj +<< +/Filter /FlateDecode +/Length 4918 +>> +stream +x]]q}_VXI~0p#00ӻc# C~NEDZꞹw{iUX<<,RjJ~/9=cpOOj^_I(?7V.N^'8M .yӟse}6Shrv)FUJi+b4>'41gI3|tƭUfΘ$̰.jVeav +f죠 +vh!9kSӃ3z +),"t:Ig^K<|tk甤-Φ(l*Q[LVEb,l-*(-鲝|NΉ J1uF'kzuQ-A,`H: [~$xۘ0Z nt-|L-*0&%LI$`\rIbb 'rx* X[ X [F-6; -H q8+q:E u*k ,)$. \$.`\r,qZp%.`]냑vKA ޵N+ xZu8Yk8.:A`e2)) x$$.p]18a Z xx4A+#qgQb4x9- X 0Cg(l2K$.]1 cl5锃OIGx\Cx.&*+q> 3wL-d%q"nF qI+@WI$.]ВD @'8T%.]孕 wC#`MF5a +c BҁwFK:R1K\4mJ\" LN!)$.YAOԅ ^₨ ,Q[\t Ih`5Fh` B LY#p RL6UI̔"Z%t F7a*`-5% z @K 贓 f؂"q)DN ҅Nk -%.H&a%SL$& +%.0JL)` hB-X`e-; R2L[2ތd[I\U(L4Rd4NKZo& UXob'l7EMY$7N.9Zo& f`^Hތvxg0"2 MD^x}D|&+8c@seg@F7 `&V)VJ6P_cn 'E93Ejdưpۚ'ɮc/|oNYJj +dtG|A+G1w#5hqt##1ѿrN#7E2s`RâCsf7<&GD>h*^ody>Vۺ5SR-`kiv@Y> +ND1il6ʈו[$`⺷.C4N㟿˿O_N'|w_w|=k:C,h33<׆tkIGy˞\+z~^Di_u`ٖeiL/8%dJ|~ʎeʾؿ;u]@Dy|uie5md'p/y6@aw QGz_7| TF9A/.c{:P/;w<询 zeZsFʱ鐺cex\^}Z}C9k1湵yamԦBgy6ĺxvg⼏G2OZ܍Z6@{0ӡv9r֑Y.o=wq&|Yy:ĩ1pl4dzc{0ϳ<:ְjDW:uﳡb(T簊6Q潈߳'nD~`#6Ps<Jw+B\ᩞ~eR$Ud:z[x*7!XC C>enSlr@{<ZoǮޖ(Y/p(͸JAjU<'pe~m< z@y:$ߎq]zʞmqa-kR9VbTц]iyL#c]ރyŞ,kXqFeԝn񈿖I.17y^wVR4ǙQhg$L4?,㡂_)=> ?Xޮ߅o|=7c1bGQvW7.6˷f'z]sX6cCCkv&Oa6wi=kmz%OB}2 yY;;\&=f^*p큻h +65$7e@ݬTM_͇anW2-u~!-}=2Of!n)rw?Ήͥ"?W>7}W 3N.o6V42uwNСdf/1=`.%K>=˂k'lVä2;TXot͓Bm/K$D{r0e>ry[kw=d+&vCa;[N{/򢫬%Al/i^i^#p&:HڨI߹5d-ay[uv,{9}h-.8F4ls+uUN. SUJզ0gn)-Nb)oӞJsVi:hɶs8:SX2 s7)mv˺l{fpݐ#SճUʇs3:HO ѩ5[^RG.+YԽ^cr֪镾 P}+WΖؤ luSNDj +WKAo7EpI?)J +endstream +endobj +511 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 510 0 R +/Resources 4 0 R +/Annots [ 512 0 R ] +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +512 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 123.873057 101.433071 105.123057 ] +/BS << +/W 0 +>> +/Dest (cb24-1) +>> +endobj +513 0 obj +<< +/Filter /FlateDecode +/Length 4711 +>> +stream +x]ۊ$}gJyH`d X#l$?}"RYQjKTɓQٕ,!9_~9zRER_oR?X]_HUޝ_Nl~iyVCNI{#|^XFR3*ku#e*3L S T`Tfh㥞"P^N3dQI;aVNSrp!f*Zh,lQV[([w` K(H@=`*x9PE:%9"X8&\V[t4h[a2 5@ tkDhq\A0Sp\:8qJ N 4Hs\:qj` qq2h xWK9.0] #qzuʠB +p\`c +`u~𮲸n*+r(Gxbyw`9.|A-wenji CtX >V̸E+ҹqbX.3p\0* ,xWjԜ+UX.q0H8`~tY]D  -) 0 "IX$qxC9.p]LVk xW=8._-3B)8xWH0 +#K&Γ@ `/Xq\tpXpm9. buxq!8. +h8. +e%ovy[v}`` }_OU-!(ж10m sg!Srh突k7o =^Y/ALh^@aU}y}b aW]5"ɖKo/ב\qu]L>mdnKs>jAz Rvyy>dߓ~35=;>sY-xyݟ3C%;䓏6>GEѻ/Ы7܂ӿWmV-k|עɦ%o33>m4Rϓ{VrK<:3o|t{ [F$YZmn_s8Ϋq[;us-ݾߙph^{g1+#b;G̛gV9rh]'B ;hܗhh4~–sgYOݻ+#w<9{0? +6B6C2투m=ݞxd;՝y gWAM ,&G[ݗp݈>o³*z[Bkd.D>ŋzdzyЍF3fI):(3~=m}BK=]G;lk;J[}hTD;4u vOsYV)Eo_GFJGh(; nÿܻhRv a*Ŝy~nz6 +[sÅUjZM\Y(M+EkNGy +#5y痱M>SuoYiuЂF[2'PpE^Z}j6׾<׆uo7}uS+.U^GgG-Tʡ|Ѷˈ]`;vGs8Dm<%,7>gyTW; U~c;786aGkg!ekWd~_Ԟ,Ϋk%WSȻn\mG\''9^{Vonߍ3к!ֽ{"s7I>ݓ蟙M"i Hm"[{o3ᐺ ;(M򊑷/j;vs4D}dyh7-Or0 8ۙpU_i? +xP-ڴg< +l嚲j{Minñ Ƚ; qܻsq^_W};i687kݙpF . 'E'9{:+QBnۙ;tpaiheimFCǕj{N) ʢma#VJeҵbqd:a_/u9TԪl.e:]sS$(i%ɈP\6jLd[Iizߴ]q#RWl[thE[mqbc~MvyP{rc$-Kut23][[SFu-h# 5ۘ,T'B,+%lfIwVŧҟۖmFmYZ*7x[-};QS_202.RSS,mu܆ꙊRk.?)"C=^+}!&4J+ikBw Fk%` q[2tg UHP:K=$S@\zZ*ژN!Љ^G4|~ +endstream +endobj +514 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 513 0 R +/Resources 4 0 R +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +515 0 obj +<< +/Filter /FlateDecode +/Length 4924 +>> +stream +x]۪ʑ}gÑ~p c?z00{ۘ}ߟR^BJ{w]DF\JU%r zEM+S.f/V.N^'8M ׷ .y˟uu}5ShWR9;*CTO1U>:㶪3\N^R gLT fXݪ20Q;I3tQPi;QJ5N)A=Mu:v +i%]B>I:[|VZysJgSt`-&E+"l1Nt ]ItN>'D]N:䓵Ji=y`ӊ:(tR|u[e$-%xۘ0Z nt-|L-*0&%LI$`\rIbb 'rx* X[ X [F-6; -H q8+q:E u*k ,)$. \$.`\r,qZp%.`]냑vKA ޵N+ xZu8Yk8.:A`e2)) x$$.p]18a Z xx4A+#qgQb4x9- X 0Cg(l2K$.]1 cl5锃OIGx\Cx.&*+q> 3wL-d%q"nF qI+@WI$.]ВD @'8T%.]孕 wC#`MF5a +c BҁwFK:R1K\4mJ\" LN!)$.YAOԅ ^₨ ,Q[\t Ih`5Fh` B LY#p RL6UI̔"Z%t F7a*`-5% z @K 贓 f؂"q)DN ҅Nk -%.H&a%SL$& +%.0JL)` hB-X`e-; R2L[2ތd[I\U(L4Rd4NKZo& UXob'l7EMY$7N.9Zo& f`^Hތvxg0"2 MD^x}D|&+8c@seg@F7 `&V)VJ6P_cn ߢ +CĜ"52cX8Dmѓٝdr &_qyռSVjsj +dtG|6iTm, ",ld&hz4S'JAf>4AKS?3,ha@ ͠E}[% ȴXӷKf>b:맟.^/oJAӧY}&,\OuB`m(o+oeZeTZ>-C[T_mdfguV乗Qi*>}|~?9 +8 7.zsYϿswް﷞G?>/ul(Cڥ ]#UH7=! oz"+艬g^׽^֤!ϝdVVfv|l726RpZ8Jy:cC!rCI32a|ėlM7$uL=? MF=tH=juިQ?(`5[Gb{1i Г+yDZk5J_Gu23zf1Qk^Y ڐ{cyu : +j_ǡsEe؈3m͑GBifl{ '<q:g&0$U<^?1d|QAf\leBګj_fiD >'<c\+gxFtKeX%(2~{4=t+w瑧Y:GIW9Vxl:`Ll&vWJ9fvÙzP'%7kCڤ'w?)itF X`wڐN]zbϵUF }]<:<|Rf,~IGZgYLkn7zhS.eE;CkZm{jrTJ5 ]ծ[|u4bIk)I>]9qQ#G7qPpZsc}Ӷ꧑5}ޒ>2Z%g_c1tc#~^cK7~,_,x-b9|iGaVhYAc].[X|6F}Nx =r*;]>0^:e=`. { ˂v+aR֮zzDc.PۥYD{a}&'񆒽!3?d|;^Ւw=OO=u! <`y%QB}l\zᰎ19)hw Ir`ڽqӃ*G+RSw2ϓ,u9ܧ^'<8sֵzSv97npJa0Z I SCGNAy ц#6&W["Q'˵ːK,rcR>6>Qt#qΓye젏PaMY~h<\w`:sRJwaL?n"5Ǒz.!5yacŷ}Mj$Ս׸lk;|7%Oyb[F̍p<2>1Rk:Bt}yBd~su;ey3)!sy3Qe+pGGHODQԷ:WX[cB6ۺIa݉!O|8w'{r<2f&.Eaᝣc8sfGzY͔U2t1zy:މ+chÊȿ =G|m7cQP?Jnޚn=vv{lNǏ-iSy7ZaNˠثV&z4aꨫ^׵>"Gl 0Gzx&,\m9Kj;N_za޻^vt_{g!c.Lc~^{$;ʯ =Cq۰鶟/a~߁BI7љᘯͤӵX}9-U E//J颳 j'S/B s1 Wr=Pe7mI:|5TMdw. B z+֫I1Bl +:؄@{ƻbmPظj6ׅ.lsi5; M;kQջF5eF&5eI.Uv̦gFo]ɵtD[OŖ*zҡNOunH?ƈZLOVlwm$vG!\|ZVn'ҥ1UWahVW_7)1E)z7plⵓ<4q=-saK'%'mVR2&G$c- uMLVӸ U5uoP~[ UZeA3VnƝo\[kzs0_ybsj%]lTzeL )tqѷY\Gkc-'kdҌ{2OszCOw xD?.e7iܺ~&ӯTa6Lh.֩ m7U'R7M˅Mіyo[ }'j.xsp)ʊ%S:ΰUe+3 zT`a]ڮs}EW +!`yye`7{19@FMƥa$m'K|֗mαuv,{9}h­.8F4l%畺*[UZ'F*k%jS(٪['K]Wi{9^ TnZd[9co)[,Y๛ʶuuufnH葩&=cg3:Hy'DjyLNGkk*q/%5p㽒H+5;&g~Q6 |_ )gJlc:ͩN_^!.#-w,~(*-Ua~ha}2?x[K +endstream +endobj +516 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 515 0 R +/Resources 4 0 R +/Annots [ 517 0 R ] +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +517 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 423.873057 101.433071 405.123057 ] +/BS << +/W 0 +>> +/Dest (cb27-1) +>> +endobj +518 0 obj +<< +/Filter /FlateDecode +/Length 4770 +>> +stream +x]]㸑}sQ,La Co}ؿSH%Q/n[U(%w%r |IM++)E>џrq:?)fob8pȓ,賙rF??)T*~ p9K*3VaK*I* 뢶 3LNR }TZmu0D9g~ +h 錞B&FY;ג.O!$->+-'D]NnuF'kzuQ-A,`H: [$xۘ0Z nt-|L-*0&%LI$`\rIbb 'rx* X[ X [F-6; -H q8+q:E u*k ,)$. \$.`\r,qZp%.`]냑vKA ޵N+ xZu8Yk8.:A`e2)) x$$.p]18a Z xx4swWF5 Do +;[@`X PdH\:c4j) p`)e$.>`=VtL)E]NG btb-qRS ?$q̀N; b-*`*g.4l-q<]贖 oPdVAI\kI4s֙ID&Xi)ElMH\Ŷ,6uz'qAJIₔ`QF`֛Q€` <> +ơSJ€֛I\qI$qA +M,qmf ;ᢈ:) &Ib @$q 7ы!D( >$xiXo³ +#3YhřĈz̭b@{1[E1f@F0+ĨU( + p5CTp3S}[sdv%5>Ͽ~o{zU? v0 LFw{xoHLDRd! |&s7Ro,,?m>c/;@݆b!X? VXa Hc!vW*H'um|̗X ;ӯG/Nor<Ҭ| (WYΚf.߃ZeN79}d^txvzf[Zڳ8^=|~ߝtKxp.Ney N<ϛE:s7١K_ʹ\Mu]nN?_a13@uvma D.|GN]s4buTSԛVO^(&{ń^C~4r +iUBe}bqi30@zvV>(GnIQ[A}-w9]ly\FBt~eF~*Fn\ +If=nLCʩǬL+fN&Jg a ҷ8{ue!UqiSCǏW˲,r̷ӗ<7P6>WE;}$ 3($,Yǘ[FR}E8BS"D s4 r廙Ƹ]F8!e==p{Djr.$ݻwwpihՕZV87C+g+COcIGy +-y痹O,y]eL3OZ-ZarIW{-׾<׆lz*{cc<.cY9mletY0Vz9{Mgz2~UV=ܙh_wwyDmtcw{7yEwPuѰ㌬G]o/j&#uӭNp%L36|•YCW}q?ۙpU+hyhu{a3+eWZCFc&ߙp]\y'5v +;s84n?^9z+b$S[}lWcaԷñ2rz<3ѐ]}WG谟Ln`l1'b;Ir_"=)=:i>a/aׂ9bs/ya蜾 +qO;o}xݙp>$ !#ORTlϽ'pOVrBZkղvm<ί zYyzw9} +ٛkBֲ.u9x"hx{Wh,uۼ 1Dwgaz_lgAׯaQ7#Z&1Ĕ7kkHWh 3{H^<CC{3 +?Olok~9~7b~|%`1;|~Ҝcmc`;.Xۃj'z'&C&}=SRtZ9l[ˑڐN-{My>ճкYe7R-4,\vg 󖴗eX[[{fgĮ驵U/{Uode-ՄE1N![z\[oϭYZr qn)g^[7a~*r(k[f{r$uH`3 El:5t8=E[/W:/ـ MJ_]6ŀS G]I畜r?T{z^;p} ;\:'$7:v;T'SDsع>7$Vm> +endobj +520 0 obj +<< +/Filter /FlateDecode +/Length 4840 +>> +stream +x][q~_ ؀?8yc# C~ċҥ{f{s}ZD~XE)uNh1o.?_N_N8UISGw&~i\{G]HkTֈCQ%29vQy%p)q*v +0æ8̰ZGFaA* T0C%Ci)Y 9VǤ Щ91tIx]t$s)k%-KTp3cL}9Z2lZ.}{\~Wʟ9O|%ry&mMYCFԌTyэT6]Z fg&_]ԋPAgm;ϧ + >׋y:C#)`^KqDqg3}k@=kM_~/oHiZC1ÈR_?"Ӳ*]TrSu9 2}~6 6ԎLҒ,gˠ|5[os]aʛ9:T.N6(j 4\T1y.uի̗Ttt,`j.4j^WKWtH.2Aɽ&emQ:]adkFvmDZ h/_2;ʷ@KKÔc{<3zIEx]-NMSOq^ܧe&?[# ew6}gB!{ +DCm2 \O"%j$0%܄ ڳFm_W[|:o6Z8UZic"`ϜwK#XH2{<|(7x's_B,hE;g }$lV'ϻ'WGm]$j|HOOf#\gC;!Ǟ ˖^Lqy&ZX7}_|uH2jza'<z҂o+/CvQhV1riy}fƧƚ]ܖ%aT;Gsho)ʱ={SzytCt9;ϗ'<j3=;8ή)]C[#bY];Hծm /C>|~ Y\:|x~nTϟ=ϟO߆|9 ukg#y }ytrQ;qc㹏dpD$D=r2ӡ<uw)hcDvFEEAg1~^XZw==pY +~F4?kt2I.1Vs4k16J6E t)Fћ7f[ٖʢO2AC-϶N:x|w\ׁhw}uW *gccۮe>HZ_lYlKf._ê~{K\b +rCQJ7ܰ꺑3b皊HF sdNs.*YAO 0!Cfq変/ˈeYUV DDS'|jᴵtd},L޻u"Wk1]j[f0󙯦jU +}V`x(VFvBʮ&ֶأhWt }өKZ.֌wVJ(EղLgTRWRhάhI lMJ:Ft\PSŖ5o4X6^ՍiRL;־Í^ ogHl Lj׸B^hr8ʆ&Che +W<@ +)#JhlVzcP6]s*fB Ϥ'ҭk-]uyDR-\WBTlDmU0䷦=8ҋyJ+"eIѪ12(z-o;XUލgKY +j4~C~b^ B4PH;5FzM[; m|@] +endstream +endobj +521 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 520 0 R +/Resources 4 0 R +/Annots [ 522 0 R ] +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +522 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 713.673057 101.433071 694.923057 ] +/BS << +/W 0 +>> +/Dest (cb30-1) +>> +endobj +523 0 obj +<< +/Filter /FlateDecode +/Length 4786 +>> +stream +x]ێ$9}gM|Hi  ӵӻBH >'i;Y]S"ӕDOU.}VJR/ߟ~8)zꢨZ8yCI`71NoN:lOrv)FUJi+b4>'bΒ +f۪"p9yI31IP%a]v0D$GAF)QCs:IgR6YEt4)Dty +!$lYiIa)I[MQbU;Y%t%Q[TPZe;u:bO*IM+00H! +:[Yot8H:1%aVY-Rxd#Y҈:Ɵ?|z/h_iv.uXr헿y+zhk)Ӣ[8^6 CPV? a׶Zh^k?/~ߝ~a ʲLAIY;R# +8Wq|K7/ֵL,_"Ϭ0] 2tD]úѩp<x̾ޛq!==ԾþO. $ڻF~&=Xs逬7ɣD\S_HFWp٥'OcwSTJVב{!zy ֏и.A~ ~N=ƋVG/28ͭU:OckX;ztD +E s(r[#X IuZ{뿰Hu+2 qih۽0D~dH2RM;g<R͍\;V/ZƉ:G}[7Zq#: +<.ӗ]! |`C|v/Sy'{-ݱ{γ]]ȽQRj_F]62C60ּ2D!;B5{]ߠcAM[3ڲ}31ۊ͑'τg;dy:__GwVI!z~F)2;\cp(͸JIf226x^wcus\+g7τrdgl{\XԲ'rUa|=Xۺ+1}.:OC<^yQX|mKL9m%oh ;o#khg$L2߬_bD)׳j伆o}Xgï+w瑧YwsSGrv^cms`_Y)X3_4b#.`>-6%kǂm)+TO>J[VW岏Pl\lSd}?.eS׏6QkkԎk5[*VhPt4+{Ό+3۬]Z˼2n7 [`yUy@3;꜎ئn^s;7kvtpތDA8+HXkQ<DZ+dwQ{IȷcL^xϛ^)yDu9&ƥGZlH8*vhs]u7{}>MqC/eY<n9L*ӉQCB|˴bEc.ТUۥrZ=j0e1j󷥷e!SN.u>޺s-с*wx҃BVm75K,9y}V耟;A\$eP{6#)36tZfe8\$7S׸O}{)7\z9>`R˒idyga(=aAl5 :tI3@`mQu௡(*3 jkhvXC5 +sZqsS>E=8GMm90(v4U"7PT}=謇CtM]@]GX>븬-(p%_/l=J2FZzٳZO6)u93V4iCʟh!3Qu.睚(Zt~Eab)CO:_6 IڪiQvԝ v4pyZ[N*ǒjyw7)N/wC}F'!zE;VLw&44*4S(nޠe{M VXw]l\Gu u -k"k Q&J,T.VUfQI}-ĵqD Jզ٨I0U V.rr-6RJjt/. @`sDr-6*1ܘr.~;(+Bm:^4*L*s*|5HYwBsNXI.]:ƆY/PHn[I˘{C$mޤo\nd-a}b-juv,{%9}j­.8F4jK;uUF*k-jSXU;Nb);iG9Q nx-[-αߝ%[.Y൛ʶ}u5ڞmnH虩]wܙ RSvBtGq#P;T + 2KlkulwD^ 19kU=y oP +T=ZV-VP9 *_TˆPTE&:rj +endstream +endobj +524 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 523 0 R +/Resources 4 0 R +/Annots [ 525 0 R ] +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +525 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 210.123057 101.433071 191.373057 ] +/BS << +/W 0 +>> +/Dest (cb33-1) +>> +endobj +526 0 obj +<< +/Filter /FlateDecode +/Length 4688 +>> +stream +x][~_ρ~؅$ By8ӳ'K8!?lbl='̡UU>}*eK\9k gpR8s\;fD\?Gr#]_sX'Wd!X)_?EVy!T*.29'%TV0\b +f^J3Pyj +0C:)8B%8sRCJ:aE[J'>@tIJN #(]`Z)-&pA l1BkN`VQ: [U面tHN t]ntA1֤.@ǽKtB0l*RFIb %SEҁ'rUޭ)&uO1;KQzDqt 5E+ HvNQ4 aG:)("Pm)&Ph\R[T( [`Ԋ@  +ty(R\@0S\[MqJUN HQ\XIq*- +hRI` Lqؑ)wh.ʀCKq¤ ޕk 4xW: 1AEqJKFn4\R\RKKq?CKq{褢'cn +?OR\`£H]`¡Ɖ~Zf@qbR.S\gQ%+FjJ2p x~p`ߘ. ).0]В0LB cR\`ˍRX.p{HlQ^R\:t[OYe\)(x 0rM ). ^z͂s8tpHe868e:آ5 *x -JJ -p(.GMN` B4lHq5XPJy脣90 0b<0:).pP\y Ϝ 'x Aq ^z2-#4撨jJygz ,` $u4Aq'N `i-h q).H.).6|Q\8lGqAE78tS\|Sj /|S\|S<e|Q\4l1pQHl~ N|D,lAS\MOq 7ѓ!盎 wM3Q:.8a-˷?> 0z977߈-o7~åCwLcRIҤ4}Or)'}|2iˏz[O@1zɔFMtli5l7Ke.]?s}X֞l5s cu-2֯qU[:T(e0Dy i~jppPJvʓt;Xyqu;kYC]q +O0P;r )ѰJ%cl2ms5md,c} gב=imurVz(בV2-{ʒo*aaCf_!lrhv,Qe4x3b?"f}yN>o1Y.R[#նdͥKe 12KoJx++QH1s#wnYzõ3鐻Et޷f|ߙlhV~ 9#fgѡ±_mdi\iŰ`yNm?vTb<2~u7gjGnxrqgadleVH눬 VyڙtWpJ7z,KQ;;q3w⮝gЌ-Q&.%XBҷOhiPݮ٘?(۵#벵S:SUzs:FK=*UnBKZ9gow*b'P{z]jV3鐷s.:ѲڮHz>-o{$7FV\3{|y\^RTWQg{55Ki}'*~|ů4JM13P_:껔lE8+r<ŜynZ: +Zèk:u;}8J_f6g ,1g 2>ǔ.I3W;MX%]V~ƢN2' v%qBG 3g\ns/_;ֽm\ա6#."<{=N +ිm ÌwXٙl;w}-_3t$kXj؝yN_ GDm\3EVZEzs6hqF~j'F޾Hw +rG={ u9w|']O~jզ.,_AouyL q4jb~=~4O԰X0!DN+xILMG6blrB]-',wBJhB^WR'{Dܿ-W-lz]ujV9 U]@өRJe2*dI)sNv!tj^j' .tv!Mg3\NK֐jR|xB6$uqp~6@jk+ȅv1:"~G6+IЫ:o{^Rfŝ#U,3:C*&WJ!ofjͬڕUu 2f#ԹfҪ* ɼHRu WQ% Zԅ=ww}2٦ڳ LxT v~uRUS)]zvPcmJ`Bu +ME x3djؓOQ?5mVɕhViޒ2:wR%ٲl=/;2t1ϔlf.y* fpJkN*4r8vdI$4Jq'ޠTo>tK:VD(_DSSrC& zs 244ܼӳ_1< +endstream +endobj +527 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 526 0 R +/Resources 4 0 R +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +528 0 obj +<< +/Filter /FlateDecode +/Length 4917 +>> +stream +x]]q}_ 5py)J$KTIixӪGxxX(*_rzJ)ஷ/\¿buQoW\!$qٛo7\o?'41gI3|tmUfΘ$̰.jUeav +f죠 +vh!9kSӃ3z +),"t:Ig^K<|tk甤-Φ(l*Q[LVEb,l-*(-鲝|NΉ J1muF'kzuQ-A,`H: [~$xۘ0Z nt-|L-*0&%LI$`\rIbb 'rx* X[ X [F-6; -H q8+q:E u*k ,)$. \$.`\r,qZp%.`]냑vKA ޵N+ xZu8Yk8.:A`e2)) x$$.p]18a Z xx4A+#qgQb4x9- X 0Cg(l2K$.]1 cl5锃OIGx\Cx.&*+q> 3wL-d%q"nF qI+@WI$.]ВD @'8T%.]孕 wC#`MF5a +c BҁwFK:R1K\4mJ\" LN!)$.YAOԅ ^₨ ,Q[\t Ih`5Fh` B LY#p RL6UI̔"Z%t F7a*`-5% z @K 贓 f؂"q)DN ҅Nk -%.H&a%SL$& +%.0JL)` hB-X`e-; R2L[2ތd[I\U(L4Rd4NKZo& UXob'l7EMY$7N.9Zo& f`^Hތvxg0"2 MD^x}D|&+8c@seg@F7 `&V)VJ6P_cn ߢ +CĜ"52cX8Dmѓٝd?z &_SV4!00}k41`CژYDlYuae+L%c+@SKR2+Cd^[$Jf_HlO\Y}mho[*a/Vbdg+Vhh|aD6V*FdVze]ސ{cyu :XKEe؈3mڙmifAu#\C;ہ87!z~wRd9 uM_qn2[)b㔽W2pe~m<d.>|ZۺgZֱ4fhcu!@mCR]1v>0} 1ůaQuG\&1ĔcG12bFi 3hg$L4_P/qFkz4݈l5ߕGfk$GJX逩3ju5F(;/iw6RnRu2XOKjxڤo?Dٞ5bYg/a6g紞sm|_g4aKY%v-.J3zݽ漤٫ܘ\[~\XRc,[)v)VY{SKoӝI̥v%O[l nSQm9ji#;lmYmf^l#A9 7__T2WkjcӸeI+߰>syRmxbRmKIyf˒6uhS<&r, ~LAh\qHpf'֗nḠ~kW;oOwmsMʘ0zqfv{̺Ӊ>u ʮ}0W?UvƑTz,u;'.uۊ?V0 ~Y8He~~lN]v)gpUu X e29yn\ux` W:R|jq2ϻn{r7{m*qQ(ln ZΝ[at\t +A%.Pۥr:ў{}GIKP;ʹw=dcK3ڢ=vt opΖVKM7-'َXZ5>r7\e#K̐NE798(Obpcb)CÇ35.y6G0:bYm_oEgg^4_,!_y͊ڔQvE:yelP xbn -%znqi;HqtH|}nm? c;HZWx=k;|6K3toGkBwӰp<2!ZGJc=<G KAa7.bQx.ϳ0[ϰ[i{G`m5!u>Y;G@ ouuA g6ۺIa݉!O9w'{r<`5f&,Eaᝣ8sf>U2lv,F(JZAyhˑ&. tD~ ɪ?T^'pݕ9bjIYy)T jF<cy+feSam\By.U!M MCWzaRM8V_wrR-oW2_zZvtD/Il"N^tۅ޲sa;b.U+V@qcJ'|xQ#x}Fos&osł9ފjҋT&\f +m^i{=] VXu]l\GӛMB6ӹ4D֝GQ&Jص]2ruk$nAE;tfӳ^TZ\ EL[OŖ*zҡnSs #}#֏k1=GZ%.޵oԾsz\hYMTKT]YeF +ߤ7b*lÆ c:0v=^SƠiw2i52mb do=_8zv^W[o.S>%_Y}JP\ +sle&W\O;b,KuO*+`I[~ vWWF{x}s/&:HڨI߸5d-a}[{eس-Cnu1ʤg[x^UuhtVb6%[uKiq[LѥW=IKEK@9֩ŒII]Oil[[]g+k놄oR3v<îk1Q;T +3Klk<{%WkBwLZ5}lU_OΖؤ luS%Je߲lHKE4KAozJﳡ#~}R~W +endstream +endobj +529 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 528 0 R +/Resources 4 0 R +/Annots [ 530 0 R ] +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +530 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 450.123057 101.433071 431.373057 ] +/BS << +/W 0 +>> +/Dest (cb36-1) +>> +endobj +531 0 obj +<< +/Filter /FlateDecode +/Length 4900 +>> +stream +x]M丑ȳ, /|b)Fc%!f(̪ng/܅JET )%}VNR/?~9)z꠨og\!$qٛo'7o?:lHS VPy?h|TAObΒ +f茻VEr +f8cJ0ú* T0CgVCD QY#负=H:ɢ.BtN!:S'I`JK:[vNI:l.uŤh%]-]-@WuE%]9QS)kQ~Z%鴞 ,h׺$.]봒wEMQ[` J\V(qb 8 +III\&)'q%@ sJ\ƣ 5^ x8$.P&.(lJ\:c%.@cAX"qO`I$.]>Q'1¼(O:,qbx.3I\ `?QtL)E]NG btb-qRS¤?Q$q̀N; b-*`)Ad.4lA-q"]贖 ؂hPdvAI\{I sID&i)ElMH\fK\6:b %C$qAJ(#qI(qAVYd @)%qA M$.f [&x¼6~3J\lQDlAДy N~$l1I\CfaŔ@f]L4H\7J{{LLb@z=V1eX3P _#Ie +b*ZiE<8a>U"!o{^ncןJhC|i`2}ktH,!#bFjt##ѿ0rzNϧDŽ3y9a:Z|g4N6͆ C\ܪo?Ŋ9X/vƟ?|0zSʅ|כ/@m2ϘGhu9j;"*xGO>.nKeJ<=Ovcsc\YQύVaxoߏzac9#~?O85KtZ谙;"=V=VdRDvJ7%&v]>jkk2fqJ۵ ~hOe +/ӶsraGY{9`gvF7֞g :F}O9ꎲ⽑\j{H4O, ߳c^㰶ȹ6l; " a a<M+O#JXk,C UqmSCǏ3 +댨-~wU~YeҎ_x2se,z( a6~}<WHV' `Ͻg&oD{(=VP_r0o[a%r7 1Gmyxz`g9åaT7jMov ,&;CXQèzKeis>?W$ӊ~AnXl@uҌ~q|tډ1]>3.U>G11X>[vY0]v`cuQ&B#Ʒ>U:OQ?w2va@`gC1gXlڸdQ~b}v~Gw]6ڛٸ XJ)dmth݉!Aj]p%`8:Qs!W>rcw{wyewPu<RwaG1N`˚m wHP"<"wbU]W }y_:φ`~7r +b@Y^v5d[=:OޝuBWC=VƨDoj@yǃu;WN/=uE쀄-ˆ@n?p!GuI}'޾?i,ù\:φ\,~m(.Xޓ" t}݇?ynԲb܃uͷcn=}jv[jYV9]<{@#d}<:φ{2U7ySs޺6c҃;u˹~H\tԲqxq̖{ }3q,>=z`Cν]]U99ly/lѦeLe~> >p_:Oҝ +vweǽ|ASߜ9oAd1rSyJޓy6ƃוC1mn]S1:O۱k%{5ZU +eot_.W]3B5{]׿Bǂ-ԙwqVl<57t$ GCRGRTwe'ٞ[(mjueVƵ8.7'u.>V|;Ƶ~SzvƶŅK-zu]αso,Z^c6Vw`)KqXNTrJq2N1'ЎGr-jUvA <>|׏P}_ѣly|bAA_jҋ^ۖ&ff{ bmkbb=^ц.ta3feMd%pZ;JD),6UGר&̨ܤlqOrߩڴ3y^1 Skr-'.'ҚoTUsnJMmمEC戲ZLI5܎s%k3N> +} ++вv8.',͟05^X@Tᛔ|ЮA^:ҏIhn6z<4qh{Y6\:ƆY>^:)[g =2U*=2w:CA*UJ&D*,NGkly J!CI2R#w׹+Έ\ ?19kUByل7(5U_6/WJ} :ͩLPypSB0V^ n)MӨ|a2ya>)? +endstream +endobj +532 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 531 0 R +/Resources 4 0 R +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +533 0 obj +<< +/Filter /FlateDecode +/Length 4820 +>> +stream +x][ȑ~_Qφye`  g1هumEJyTu5>]K(/#B*I_~'bLS/dU+ +Gw6 !9}˟u!%oGVB A9AR +*.* JI*ADAaZ` $AF)QC5Ncн3z1$t:Igi%].J:-.)-lqHI:B6IaV:bb.I.+:آҒ.H$t*UNVaEa|tx咨-*#,l!H:!Fa^VY-l1&-|-*x%L%C h%01 h$7OA4tFKD`lq%&V-6ZbJ\`H\`AoIi/qL%qI kX.{uF ڵkI+ k-j:K\vVZ22`VK\ &F%qwMT$qwM €5h%. q8i5dQ78- X Cghl0pK$.p] kl5蘼#OԥAn1ʃ>I\ب F;j I\8 ]QbwAKFƒ`QIK\2P RASҍV6]=7L>3PAcm·y F>#V_ڜ6C'{o>]c_;l x O_~PXJ_E/9|Icae>&EYqlnF87 c9eu#4F{XfJ'mlȳΌz=ձP1~?2n6G3lK;<mhl>N0]Wکd6ߏ<7{áLvP ~KVml^*g^.XxC5&"`a !b^E%Fqh꡶ G@F?çj+/;ͫ6SSdc$lv7A%r&@Ob-! -ףV8hQmUaΤ=tFyry1BVE"g/f>[Uż9A>ϣ@ uK&/qf"g7u& +[B.p>q:li^Wm&3|)靬kpܧ0VOsl9Wv4ʌ_.V<*d-mTt\ڪ^u̼(oa]N:%XŠ!j_,;8NE:φP Ahq!MQcQǟ(e;ee;>âP@ue< ʚl'^*&v(+B~>k̏ $F_UluȬhB;k|z<~k6H=WwPB.;]Ԛ3v6GόX!uuoseeqՉ+i:;f6VnQEyK4&2SCO2y6d:GJzg #[Լf|?rfP<#f'< +ufCgti u}"3sA2#yKm[g<)dbPsuy>r&|Wfvm|ڏ-xGc?Gfćf:v͐T4| x<zʓ:]Բw=ZUmF^&Yi<=ܖaT;G:φXo>ޘx7YyZOy:noշf??YмBڝ+bvVF*3̫-%b9ڶƑ[Wv>)MQ|{>y:4+duw\T/ׯwfw7R+>'WWfG%fV/^k\]2WOy6dwA]Ǐ Q(#vړuw[p%mu.]gb=}Hչo k'<B'U`C.CfD_M\KhR{Eqŵwm̮^4.:K۾tZI ;ⳏsu~ߙ=Bkۻ r__Ŋ@iZnŪ:B|yγ!/\_ķ.rw]ޑ]54}<-_": Q:2A["dCA>tݓ]sanSEǗWO̷ӷylJ ]lDtӡyt(e;IY{W{VJGhQNy:g*"X";#꽘3?/QвL/ςݭlOpq1k5G87o; g9b0{%wp&e6գV~ˢ/yڮl)u∼|y:Yua=:\5~ʇXر<.LR϶&LWfݑ'<fdyG'<Z杻gfZQp| uγ!%\}_7޸C{y_ԙɫ%Ո)nsY uk5>Yz}!{ d''>kfLR[tH=Ȇ=gTM򁙷_^lx!7Yu7=W +o{'<yV:7r X%MN9 +?M, 3͏{jayՋG}Οw&]E6^ ?W|&"'~ay)OD1LXB:'?ɏ('o}Wٔ.S$2d\~yeg칔i9ϏA~w{HW~vTt&V_7j7>\K)~/?8?ih\9J5~RG  Bz-Wң8WS!UlkU<k:[ U]*h~H~:߄~t7 t otk7QʯZ)M5lgTRWJ :Ovߩ4a25~`(HT˥RN.b5Wg*RUMe) 6ցdy&i෸ukDq.<eUpcVQ]MvqA(WHkڭ^',"}#U*e4UH#Qf:$våCKG%gҭͤݐuyLRO9EXM'4~ʺ5{)+텮a'6Fnd܎ȼwnچ=rre~A +06m[kG\mВ*o]Y63wR~*,tnڦMY>\` ;M3{^^d_VK+W[/'JxC??{+.PmvkV'-Zh'TR:-E TCOʰ3(S4vAtUYnx[r=C;;JV_2 +2׮Rj%myevA> +endobj +535 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 728.673057 101.433071 709.923057 ] +/BS << +/W 0 +>> +/Dest (cb39-1) +>> +endobj +536 0 obj +<< +/Filter /FlateDecode +/Length 4862 +>> +stream +x]ێ$9}gM|Hi  5ۻBH >'i;ӳ[ ӕagK_~/9=cp.j^_I(?7V.N^{8M ח .y˟v:)`R9;*CTO1U>:㶪3\N^R gLT fXݪ20Q;I3tQPi;QJ5N)A=Mu:v +i%]B>I:[|VZysJgSt`-&E+"l1Nt ]ItN>'D]N:䓵Ji=y`ӊ:(tR|u[e$-?">VI1P^U2P^VQ3P _Z#0+]LPj%n(/1yoQ!bL1,Ny^~]VvFPR;W&=o@1 *1Us7Km62 n?ld,rc%h!kSJwXcE`h춾:]#b|mlk;IyVL`XzzE#%~gbf֋1cLlm#%S'PS>Sٿf&2;GDfC3(hkV2]%31h_O@__iiw!4vݖN뗼>oeTNҙ~oK]u]<$͠r`oթ/R%o  CD{]~a 2I +h;,/Ŝ8Mr?0麘W8iӳl$=P&Oݝa|ru+O-Q U2HdG.VG[{LO$t!{mq,Zar٤C2$LcpuǜG_&!t25ErkDP $g|NKOlD5xފ`qyi(=1D~ބG;HdC@8V/ZƉ:G}[VP7'Zq0Gy|^iiwwhdCxvoV˗>?yإݨEFyC{2ϣǸ*QMG[\·q0 k*kZyM<#y}\z2áNs٦:@ftcu:_na<&=:yH~ms98|?iFcaE}ŷ^ИH]p=i75Zz8tgCQOgVVfv|-nޡpHeGݺdZwExFtKigNYM/j~y8,vmOtԙm;Mjw/110,Wq Bh-_&oo)it^ ~'L0ZɻsZOT[6_oi0z12*2˵Y5Z*#5f}kW=|ʥts/Yj +ݫV˖*GG4l)Z=J7wP-uZʙg2(&?pE|sm+bO[ڟ-+r- yf#,u9Ｃg6inúow;aH3+b-s[viӎrUs;]`fS`m~Ѭ}Q"?3͡觙zY]&Rm7of3PK缵E3LfY<=s.Ѽ5mYPJKĺumq5ǍQY~ +hEGNynjSJrˍE&v+aR*7zm fZp.SUBۇ){?dۂP{CW|;Dzgz: jy4rףg(/DTP UB@rlQ9Vtxk=v2ϣ!*k}ͣkaN_ ۹%>{[w:$z`vl ߽tOy\y;(챟fRB1 +y4ju.`rLoOΓg?6ۺ"JaJ8aѣǠad ow܁W]mW@:Et5@QtҞwM]bvoGL] at7UᲉ^nLYnusyj_7=ＬMJ6%"u9%X4iC_#p ngbҩIYk7X}Y|ᇖj>EG4Œ_tv,g"N^t˅wع~{񎘄Grݭ3NyLիZ>[?_[C}Fg:Ur\ DΠbj"H]B36[x^`xWbB*V]Wt^ms]f:wf_ӚȺHxZ;J_D)z^5,3*7)[Ov_ZCg6=5:;v +$N.b4z*Tgݔuٍ*l([?tJ 7f#]k#_.> +} ++вr8. +CM +ߤA&͸w2 I V<#%;F50[Wb>b^*lÆ c:0v=^SƠiw2i52mb d>o=_83LWpuOWV$(iҡue/-[I4דw #v,/J+X_=o^ɡ6jo.m #i#QiinՐV'?. +endstream +endobj +537 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 536 0 R +/Resources 4 0 R +/Annots [ 538 0 R ] +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +538 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 292.623057 101.433071 273.873057 ] +/BS << +/W 0 +>> +/Dest (cb42-1) +>> +endobj +539 0 obj +<< +/Filter /FlateDecode +/Length 4739 +>> +stream +x][~_~`! 8!ٶ&<+ґTRώ5zuN]QIW/Z0zt™3A>\׎7OQsHg/_.iq,+STiŜwBB*csXBe)@`qZ T0CK J;3R #T!9'u0k$Y4tR0 A'tJ10f5ib#hEYIl)J`DBa Il J3kMt;?InJqJ'3"uQ$-@`Q\R:[4(x(=1Ze݊+A liR[H [`KHRS$ bkOiE2mp(."aіb%`%uEMq (.P +H(.P`[ P\@Aqj., + 39 +(P.(.P\@t1 +(ЮR\J Nq*^Q\JpGqbI Ы 8+Lz).]鹦@w@Th48jJ%+( Io +1;[@`N* PpK(.0]0clńG-N[2 xwAp<Ì*).0]0SS:𮐁S\` p(.tI]` ,ơ <֛RS\qכ₠8֛X6(כ₠aB` @[p֛`'Jga `m\oz f` tdkҁwZ p&<[jmA?>8=1^ pp/f+G 87Ƞ[2jiJɰ3NGLI >9J2WO|?)KZÿ0m} h,GD#\(F +tFbaC*W0߫X%u Vs,0ZQAFvWr=I :Zq|̗X ki>]o|ᯀч?MxlXr~c~P1i-i)U.qBlhԯƼbz./ԟXӞC.0(p<],y{S~.P?TF5L`M@~I5y26es5uɰK5;odoia%iƨ^v9`ZEB/![H˽!rZK=SW!K=#>$oTLu?ROܾzMA;# ǒ9:}PY@hH\Mu9B7}fl4Depz,!Z2zTgb1}{^lm"w,:SK_s:nFK=*?}W?cH˺K~aTNJc6 ;L%=[AQCfuَRe-'׷ꘀڔKC9fӷ<7P֎2,]:}}օvMRZ߉*_##yW#|D֎ٿ*%ؙtص;}5GԆ?lD: )fkWlv_TO\vMkWWW_K+*v`[G\xו=ը{;?-3'tO|?f6- lɸ6j}yNԍh8#?#o/j;;}w<ywlD: vW}Zvz{mςVWZ{s:n.W5okq⎝yN ta7S&${S]q%3pY_pr܏ݙt{ӷFh?h۩M@3ِ׽F{dwdDVй]%Os:$?s}H~XA铧f?m:f;[Wl; н -GջBjKh eE;wܐ6#@X +y-2R7YmNlT8+pα.'y9}\EggׄNٳ5%.Le5\+rcL ^8ԩy'L0fgFPW/*@LkMAi°Japaup-<7/Íq;E~Ich294<;dpbDp/~ߩ-NÅ֘?CY&ϱxQ6N'囱ʞ SIum^(1A[τZ"poH$ׅËͩw51CYlH!Ae:(K|&Nq?y.K0n`bBRǟ!L)M:ߤl}GDjxϽbBV9835lfKN[ +,7BJhB^UWR'{D|]n:[D)tUw oTZ )oDME֕f>KBfn,#w;0J(uEE,|Ik<.e׺dSJTdm-4 +ЕBU-cV#hߔ ~ɡ_PA}:8Eʔ|QLZth2.UX613$7}_]b]AFCUtLuI6E.ꂵ+ {}4٦ڳLxT vzuiej䔡Q\ =91ZI}6&0u:&"d> +endobj +541 0 obj +<< +/Filter /FlateDecode +/Length 4922 +>> +stream +x][q~_K8``#0ӻc#8/%N"c%U")<]o?_~)89"ZQN>,a +əo7\.yL)yoYEv +1UJi+B0 *'41%I3\ C[UT0* KAۭ* 4I* J+ءR("kvG{Igc2It0NK4y]t[\RZ94tl u`%]-]-@WuEy%]KH%T q3M.Z$֓6Ì l%Q[UFYBt gCl5I\X 'F+j I\8LX&]Q&bwAKF“`Q&IK\2P xSzN~]H&s}yaѺbo-'̓P6|]p(uvNu2XTAuQ/U[IӦM#H BŲ6;L +s9cy)78nWV22Ye6E[]9GŬ +sNwVeLfbzިT]\-TۈNp;FmT-'hU۵gV;vYoyEgĶ2l~f1 ʝ_lrSϥl6?.c}wvd4,0K]aΛ/:tnCmvݍ}S1; B)s[viӝbpeNe}0Cwm+[f^Eev)43R++VYz)9oic%#ˮv柞{z޹9 sw)m~im]I,>]4ul^;9qt, S< +C?ks^EFxgep؂vN<w+L9߼*,4aSIJK46M"%ףNE{r;?%Fʝݎceq=dCD伽C}pL:VΫMT֓:vqOXy=O`Eq.-[謋${n흎#h9w[牟I|v4;G p$k>rn㽓r"8oK7+ ->I|>*J²weEPDSJ.=T"]Qw<{(\kn˿au.CoEn'SoߴM/l2 + 1$y5CٽͶӋdd4eezT[b$py3˞C}dVia7~-+6+Ej0#\CjC !vЩ-grͰya݀ b\/QQfg8ʶ%,yҝ[;jo]$sg.2SC~{'<2ڋ#k%->!}tK_gZ5nύ(#g&3bvQx2ӡXGI0nnA}"Ƒe˜u'V ڬHۺIa|H'38wC#=A_7g†w#vsd>4zJz;Q)h}t=+KV^:l]_+F6F^Yi!<$p$wޒ0;G<φذo>јWnϵ<ßtQ|g.h=}wL'<ލ?zAHsEwlq:"W"{8rlǭOgX(]G[>"uA?쎚!N?y)Ϡ}~O m6 ~i}y6e,9zV4u{g#y}ͣ{$E)w^~?x<|W#4EF('<J˳QGr6FdgqTT}s5 +ڮu7GOkF'wPgkpq@,0ܱz'g>j9_|cbIK/:O'r,\4ۅOd'ALԎ8pjYEv-))g+dHD67W,뭘[&T8WSe*lR2Um {4Z7a5: IkB7{Q&J,RT]vF*u%e$nA*EQIga,@.tv-:R-k*=LiPSQn7#ʖk1RZ7f#]ՙoԮqrzvVqT /Mʻ'L"*\2*Lv*YN徲I=)]w*f gҭk]uyLRV a &n4eVv/ )$V寗 {/K_i~y7ܺ+H<)o):m:Xǖ4uܘeWKJ]TWm/t;5v=v#ѱi |iKd>ڶX9Mgے|'mK:d eWajd@Ԯg +uu-}E[% l`}ie`ׇ}1ٗJUKkXZ O|חuͱP2y[ղl7lurh.zvز- \_)٪ҲPTLY-1tKnq{)TIGvGoreūޒZTKY}(|JEu5!yes̝9ϰ *?y'Dv7jME!_(v[#+& D֪dޠHNJ]|r&=VP9 _wP *Yvz#-LPZ>4&Zm`У +endstream +endobj +542 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 541 0 R +/Resources 4 0 R +/Annots [ 543 0 R ] +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +543 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 573.873057 101.433071 555.123057 ] +/BS << +/W 0 +>> +/Dest (cb45-1) +>> +endobj +544 0 obj +<< +/Filter /FlateDecode +/Length 4929 +>> +stream +x]]$}_QφMa ?b.ݵ6װʔʪ陭5+# +YJ}VFR/?~<)z%ꢨZ8yCH`71OoOrg"!p )_s5 ?R"M<=LP"ܚy6D1ά;Vָy[_s|?/Fy:߈U?;hlbW}Ōς8pK[ky7bכzϨֆ1*|tahhua?RYSWHzW~Ykñ2r`gC~^ 4mP?_?'޾0iLss{<rům')BE@u@eGh]%?yͨEF{ =Ǹw:G:rv5 >1;rg2?{@WhΗ;{``Cߍ8ߑ٩Jܹvc҃+yɟDG0Nosxg3qzY|zeSlrA#7yk빷5JVjjW3.{!QKկLtqGuot,){;{ ud;Fy`#k'gody:__GwVI!z~z)SIg|^GQq! +ieVxߗ `gCrZa='=;cZyܗs=?Ĩ WFcnF<~=YPװʨ7#~.]bʱkw/y;+])Ll43[P/qF˞toX]+?_V{%`Ǧ7cĶ~po/110o\;`moGPODddDHڤobCCkvtoa6ē紞赶nȗwQ/,@Dž';:zy[rN$5=܎KSEkbkiT&5EYպHaiNX#)dKg.KyiZ|rwrm9n)+\#igmq־y aA9PvpSJ͎Kj_f}Mm,}rUgF؟g~6]}Nϰ-}}-O^+i"5J(Yo}g s|8_U՞Ň-x{,bfBݫ z7%L^d,Z^T|:t#XX:,\RB1ƚ'z] 1 t[>WuZ>56͒jak6:F֩ȖT!rSդ1%Mv +qx^`xSbB*V]WtDxs]f:wf_ӚȺrBvRzD-zר&̨ܤlZ? I~jY*\Kk:At9|R5WwS:mjdVl([?ŴZ%^N޵'Ծsz\hYMTKɌ' *Î*|5HсWBcNXI.]:ƆY/Hn[I˘ѫA1 +۰aBwN=hh1]Ϯ:ה1nZ.݇Llt32x27}ž/mm-j.xsp)̊%S:ΰUe+3 zT`a]ڮs}EWYiK ^ӼҼG0 i&}06“\eslYX=ʲlC eԳ- > +endobj +546 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 127.623057 101.433071 108.873057 ] +/BS << +/W 0 +>> +/Dest (cb48-1) +>> +endobj +547 0 obj +<< +/Filter /FlateDecode +/Length 4833 +>> +stream +x]ێ}gyH8 lﮍ`C~?("tٵ;V/%X*!蝹~zRER_oR?X]_HU]/l?_~yyUCNI{=|^XFR3*ku#e*3L S T`Tfh^"P^N3dQI;aVNSrp!f*Zh,lQV[([w` K(H@=`*x9PE:%9"X8&\V[t4h[a2 5@ tkDhq\A0Sp\:8qJ N 4Hs\:qj` qq2h xWK9.0] #qzuʠB +p\`c +`u~𮲸n*+r(Gxbyw`9.|A-wenji CtX >V̸E+ҹqbX.3p\0* ,xWjԜ+UX.q0H8`~tY]D  -) 0 "IX$qxC9.p]LVk xW=8._-3B)8xWH0 +#K&Γ@ `/Xq\tpXpm9. buxq!8. +h8. +e%ovy[v}`` kv/oh?!\=Z1]M&Yޚ<yŇybC_,a橱ֲ[mlm)۬֙mI6>cc[gϥl۵ksO}.~3% J/X[Ƥ߂n>oz} <vVok~ ~tm`{|ۤ񾱵iN-3Y$]jOӯsYjӤQM7XHuȮ~ߕ໺;.N̻n9+ oX<òvܿʎϓ}lbHf~'~灂G"e8L4Pj^|ҏGZ1\]m(d)r|۾q&?"L>$/0uį%?&ޗINǮeJs[6ce5=v_6&|6qqL[SMXcEn]T|ʦ +Lh&z9 zNY8zAPikVG{ҼNr&}}7˴󰽗2]d1[B=5+/׹2H2+bAsU:*…grKݞ0 MAY~=潖ҼpzGq>^v M2t9fۥm#+5nm|RzUȴ} vۏ] +p,ꙗ̝\=f-B@hd-{Ѽ#>s:d:;P)Rm iŻXK*+&wXbہ]ONrܿ`ӡuk{ݱ''tOb?f6}oUl}`!u'pQ<#o_ll嚲j{MwqJ&qY_>F oib3r%UDjeRJIGka':8qo/iSڪNK:H~}19H1OuR HFrwsCzyc%ˈH5;x[1hz]E} *͉E$() T.=-mLi/0Җ,|~F +endstream +endobj +548 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 547 0 R +/Resources 4 0 R +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +549 0 obj +<< +/Filter /FlateDecode +/Length 4894 +>> +stream +x]ێ丑}ϯg#~ c`w/`؇)F}HRY]= Օ%<< R_~/9=cpۏ.j^_I(?7V.N^G8M ׷ .y˟~uu}5Shrv)FUJi+b4>'41gI3|tmUfΘ$̰.jUeav +f죠 +vh!9kSӃ3z +),"t:Ig^K<|tk甤-Φ(l*Q[LVEb,l-*(-鲝|NΉ J1muF'kzuQ-A,`H: [~$xۘ0Z nt-|L-*0&%LI$`\rIbb 'rx* X[ X [F-6; -H q8+q:E u*k ,)$. \$.`\r,qZp%.`]냑vKA ޵N+ xZu8Yk8.:A`e2)) x$$.p]18a Z xx4A+#qgQb4x9- X 0Cg(l2K$.]1 cl5锃OIGx\Cx.&*+q> 3wL-d%q"nF qI+@WI$.]ВD @'8T%.]孕 wC#`MF5a +c BҁwFK:R1K\4mJ\" LN!)$.YAOԅ ^₨ ,Q[\t Ih`5Fh` B LY#p RL6UI̔"Z%t F7a*`-5% z @K 贓 f؂"q)DN ҅Nk -%.H&a%SL$& +%.0JL)` hB-X`e-; R2L[2ތd[I\U(L4Rd4NKZo& UXob'l7EMY$7N.9Zo& f`^Hތvxg0"2 MD^x}D|&+8c@seg@F7 `&V)VJ6P_cn ߢ +CĜ"52cX8DmѓٝdrgL׿Weh%C|a`2#h-`CژYDlYuae+L%c+@`,{J` ͒և59"&A˶vz$[F1y!Yy7vq[10h<lm#%S'ŤDi~fY>`a@ ͠E}[%ݧ tbާo|tO?^~?}?^w_\_0bEvï3=}KAfy," i^"?5C;6W*,m?jKm.=w'HG乻t#W6[2Sq7D2IONy:$߉c9z.g~G̨ul4/dPH=l51UKC؜6yfamfGv\FF*C8ֵRwF ]u5->V8siWc3O~r0,\mIC_9Ɗ7Sse^Lugncֽkf]D1 -wiӃȧyusKOy/vRW9LZ6ϝ[aPanMTwK46O>" +%{]*d s/S3)7ILyىx纑n-3X<=}ׅ(^n7k>.Vcx@x'a{ϟtۀby2wH/:] ;un+d'Y6qͼP>w8`ztAēy<.9pBr֏c\grRl4zp#8S|q^>lym4nv{8$jVӤ5GQOy: G?g4Z69 |8mG Xμ2dܾZnyG&)-o9F۹y[W8}JK +vmfɓwfkArA֢G/zm˛1b/lkxj}U/Z9(Zr'$ayQ=^fōFBXP~Cwwrgi~T?תiW,$V~=gnY\ڰ鶟.ES7)UZ7K`iݤcu'gZ;+ƚ^aZ_,*Eg7}Q򪈓]vpع~s1 d7jћZ>(?_[C}F}t&͇sł9ފjҋ!Mv +:Rn^i{=] VXu]l\G;}B6ӹ4D֝GQ&J]2ruk$nAE;tfӳ^TZ\ EL[OŖ*zҡNnH_:ƈZL㵾Vlwm$vG!\|ZVn'ҥ6Ь2z;)?SoRbA)zplⵓ<4q=-saK'%'mVR2&G$c- uMLVӸ U5uoP~[ UZeA3VnƝo\[kz]t0_2QLO.6v2&솁[ة?e4.iXo~`5}]P_;F50[Wb>w夓b*lÆ c:0v=^SƠiw2i52mb do=_8^\o.S>%_Y}JP\ +sle&W\O;b,KuO*+`I[~ vWWF{x}u/&:HڨI߸5d-a}[n{eس-Cnu1ʤg[x^UuhtVb6%[uKiq[LѥW=IKEK@9֩ŒII]Oil[[]g+k놄oR3v<î 1Q;;U +kcKlk<{%WkBwLZ5}{lUYRΖؤ luSaelHKE@KAo/9Jϼn]mr +endstream +endobj +550 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 549 0 R +/Resources 4 0 R +/Annots [ 551 0 R ] +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +551 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 348.873057 101.433071 330.123057 ] +/BS << +/W 0 +>> +/Dest (cb51-1) +>> +endobj +552 0 obj +<< +/Filter /FlateDecode +/Length 4696 +>> +stream +x]ێ}gy)ހ $Aaϡ$^Dnݳc vJaD*d}pO_/sF_brЫǙ:#) +\05u߾{G^UU\)Hw;' +P 1sXFeeK S H\`o8 <0Cz +0C9I +f`vH%!H+F'MgQ)YT`u:Ӻ.tZ9tI$8l!E[-XlQiN`BFa YlVHNtg'bu:᝟0Z N'egMУH1:[0t8xh=[UZh lQ-h|-YЃK(: +8P^ws,|9р +#ZH@ brL%l!\V[t -HqְE@m h@nIqߒ …  ( 48W;9. ]twqjc[ 4xWxWkdui5R8 n8.@vX*/*!չ}Ps\@]ep UF( HY #Xo +1;[@`Ni PXp\`2`c4ت>X xWz4 tqߢ;imw1Pi x9.wQ+5FjNޕ* x~p$ 0uR.t"xq;$ud X.q +5+n-+ @ni8P` +N-Bqa@y ``/Xq\`=u9q :|v868e:B8.c<NN #肄N` SL@q31*t~t1ƼQ7!ß/-'W雞s,۔$_%=[Lw#)w?~6`a U,?1u]`Ti$.ULMS }K爰TD'7?˶XUV]KQD&:xyO3<5- +Fְ\1!e  {\0[o9dAۤ-uВ)OzMF;# ǒźΟ{,^#T1ԝyΆPwy({"epZ,!m  ByTgb1}{]lmʷvCk `:ov_ӡvOMmv~b)ze۫njs_s:mܤo淳δ`;cuPhy#uZAһ- /&g"N;|=:ߪcnjSFǏ +wOrȷoun%eGUrhg[ډE?UGzJGQ[Qv9J7˽.e3h#,ΊiFGteu|kkv9]|S }MT12Hv18s'0^G6G%}~G:p8aі 1x^:x|wk_kú:vEdWk=ףr[Ѷۈ~3cum3ِj7awI^1f c qVw; ՇdN;[wgӡy}t6^NFa*-TӰ ~^zs:٧#tcmg=, 7 +s:$?s}H~XA铷f?:[g}O& 5wCqĸ9ְUru0oO cعTbA\εĈ #Nq?E.dJe7BRN?C.\St51IB$R"3"sqzQwl6d3@ӖKMԢPT+bEwUg0^W?Uj5VJ㞣TJd +YjR ۉ.K&4fWwn>r1ۅM7frKʗ!|1@]\񌬯QvU:mlWQ3b("~Ǥ6+IЫ*7(x) ]3W*LFHʐ{וr7xZIf!$.\l]_ &}$ݜ:DZUY%w!I*T,38RX̕P$#p"6/q +YYrf|.f{slԊ#}:I+GrJX )qї^zkf-ɸvsJjխ,y/|Ͽ.t>m6JdP['z$+{nS U_җ&!JU|M%(ĞR̠2kk)uXU"5Gۦ áN` +.C*G1iҡ%UeA](b$T7*7"ꢣ2E'YU> &Y4La{=6(Ԟ/4_Xf㥺?> +endobj +554 0 obj +<< +/Filter /FlateDecode +/Length 4237 +>> +stream +x\ۊ$}gRyH`d XcnK#af +?}"23*gv$v<]Ul)DӇ59Zk%6M~s<;KDWS,}N]⏧|v綎;{Sk.UL.9ۨ*TEǤ3b +0b&OGU %j* *(pTUّ WcVT֪:D+:MѓΤR}Uu: L.:MWMJ)MGu.KtDV 5]UuK.u+*XlN`b-DBgK.GtΙ + +lXb^`!h:D ez^"xI &_cQVa^8lzH`PXj(рPY`tiD,DJ,%TҸ x`A BOA%xtK: K.`Z 7DP.\j\ܩqAbg X75.@%i\ո!8`G\ x785. .׸jIX % 3U `и>b z ' (ch +z8, X Chx%D5 2ԤqAﺂSuո +V6.qAb +D.h\7QvjMuZ "xqCԸ` gẪ[KѸ wAK^04.8T^5.H]Cи w-!Qu׸q tXʚFjSh:uaM޵6W xM Rf/A5.HL9i\!KQl!,U,I֦Z4.Xd,4.@D`CwQDLD䘐.yS2Z\ָ g2؊ȥL׸o8\Ҹ0sqAq"S<"sMAtsNѠdj\$LrY4.( ` S:`Mh\d;j\&U,!xN7^&Yj&aqո|ӓXo j7)붆fָD(X4Ue݂o4]_I゚E7ѫGͬwc3i:."Ӥq.o"r{ ,qz#V{[e|9GR eIU(U ,8a9}GT g̃X~XRl9ovRV_a鎈|$%uD-n ;?l8CGmE" [ I1{:OBΈy|p:i*ﴕu^ $9F:nWvka cT.ڦ4-@[aϬ "GtP&Tϻ)(yMߜ@+~\ +6Joޟ/_}ϰӛ~gS|~D߷-gHY(eko욣~:t$MG=#VNhZ_n]MT®Nc7ߜ~f-z?WcpZoqn6\貝OsZZmˠ2~7>}]y׎!l1Zpա;zu>5v&5yvׅ:}bɴ[LQp60|ni[^Qqw;?nr;y:r-(\JTXxgCxr[} oWݱz;py&Vĸ_L;_L_}?͙p:D֭gl ?[K n>ײ'xmX`tsczoX>|*LjI#<cch\&uԃgp 4hѧeax"bO }G-NϽk8}ie+d y˳ͭoNZa;,1ʳli^׿IG^a[wsv:?Q`r:߲{{tlmuroؿb^}Xy[v|D)y2 ߴ۩u|u;ll }JN|a /pm&_I<G-w׶[o>rlx4oI}LHΚaqWLxgb0^$3%FЇ 6Z'n BevXuة8{-PMLJX|KJ*jdjĜiQG܃|b +W,ԏ0DJwHQ)otۭXkRTm7pBgy*P7|3un'ܟk|[SL^`y"{5ˋ$obI^.Tw>/=TO'y#mYގ"Q/7?pZw3)M+dǥ=v6a[ n(N{m7_"tzXzŴ#!P+{-م߭5R tߜr<,r?dA'H٭/w7x6K} ipJy7מy(g>y[4Y@Oku+5۫d>Re,?p.kmߟڭP=koM=ŏ'~)bCN1 ׏vEnĢ]vAMݗDrm׏3`{c ׏? +}~XfrG1m*ń* (ld+()8~v.M.M>%t/}TT.. TK 1OrR4a2^? Kw$j/.'Z3G2n ez("̯kĆq+a̵-oAx9E+qO1+ +N# bLi.TX] U!ei go$iipgYa.Yi/WRIwHɄ<IjV 1SXJ)8\q#8XюΒGSw>Jq3w+| /mc]N۱΄p)L&uLq/Nm!z>N{JdFo.-֍XdVrrՅc'AS,7}{4I콤 V +㴡axCxYq2;7s^.C#.ppY.-,[:EE\Fq):&;d񓜊} Fe:۬l(DA^ȆN} +`!}10ӥ]s™/mrҘm. +iQvJ]o. ]6ZSZuJySt2嬤CYnx#Z +#ҝ%G,YYfeǾiwmb4Y-J^v:STi'!WI`&g׹}Vr<g%'LlvW#ihVV:@9 +4g'AE큺v 4.3^AoבeorVNb5 +endstream +endobj +555 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 554 0 R +/Resources 4 0 R +/Annots [ 556 0 R 557 0 R ] +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +556 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 536.373057 101.433071 517.623057 ] +/BS << +/W 0 +>> +/Dest (cb54-1) +>> +endobj +557 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 299.626036 101.433071 280.876036 ] +/BS << +/W 0 +>> +/Dest (cb55-1) +>> +endobj +558 0 obj +<< +/Filter /FlateDecode +/Length 4808 +>> +stream +x][ȑ~_Qφy30?beN1=>/$EYKUit)"//$}UNHz1Oן.?_¿`T]08|tןX`3_/;l_?u )y?b VP97`T^8Ŕ$p ݪ̠dLTfX +ު0A: (%`"Fi7x tF>&D]N#| KEIG%%-N)I[ ;wN-$%qop qA+@WԥA%.p]В"Dԅ@?T$.]嬕wuF#ql5 guPL!Jc%xW$.]6%.&J\xFiH!x |Ru~ ((K +$q!%.Xc$.e4o1<-l(qlH<1ft-"t:H\ ؊E/q~`wI& B-X*`+g.4l-q<]贖 oPh"L$.XC-Hq%.v%.2A u4!q懃m'qsDl$.p%.e$.69 $[I\M4R$o(qA +&B-pe[ ehUn$a(ٸWx:vD\.#=ZF?e=ǭش]ژ|gg`@! Q&;hEw9eب맟._~Oo-gz:=ɝ=7ġɲ?BMb  + 8^f2;=mњOg&89oѡu]GMwdΜKs/Ǝ*o:gk0n[u)Sg:Kg.3>խWf7Ӝv~G]h&n{/\w> +c:qyjS ]}yA_p29'U4FmR:!_܏*&Vm8%UrЬjV[aGvl,.`gg̈́Jq[A'i\mtstVAKU8ٴ0eҹH3њ[&%:bRC"8ƉMs&5na`DQcQǟܕ2e/_ۿj>{ یhUĨ(;V$(fm.Œy_VRg36(˳B~9kc;$NLzۣnmZhc1dw}c0=^ԔG:gCxsRhJ֩[%zͶq}]7ڑWi%7omuFϟ GF5= ;,]˗ՠ}U4~̠yF6 +9 +efAgT]IHODV?-_ctE_Sdll\uBv|H|C/Oo#X!oLJg2,G}d+%#D 7#m۟#?ѝW6>62 W|[-&j-_+FnsS73 >m4V2wErk:QXas:n?}U(O\Kt 鐻2:O쯉շpм;;l6;gW1_?ֶʑ[V6>)9Qm4s64m?vza˹<3}=xUʐݠ?{o drڗddՃuNɻSPuWټ6Q"x+j9jw p:]kh=̥y,փއY7>p`!t'zQ GҘ2j ˈ&{~:Twm,^4wn޶/MsRrN/lXJFKZ]FMyVߞ{no.±5{{b-+zy_U{WH/9vwZfvqu];Rײ=bɮٌ +2A[/ X!x'[.o9˄WO̷ӯy_%{&wy4":Xt;o]k')J}')~?}d%pȈڻ`4G*emQQP^̅(Yf;.k{wksFnvpՕ:F87; g9b0{%F[_v_zT}oYe?O}pP-N7F<8_:Vtfc?Yis|<:;m}ѶcĻXys8>6c* s6Tŝżs>o^);W_#?7^C/H^VfյגJ]7Nv /|ו}Hߍ}к}C½qwϘ˵O&vyl:CN6M; +mYcVJ&?qJޔ2j|>"Xj|>yYn~˓hN̰cCdh}7r.ʹ|(J" i}~,,~v-[wڪFn鮊 e{4 bz;t\քf:tk{(WUd1*%˄P'Azߩ4a25~Q$HTʥ\N^rskT(bWS*m,/1ڬeǥ V^XҮrrz6qT/UWOXYƯ'pE3~T ůG^EV 4B- `/PI.#6CC1IXuA]3)\ +U6U+P~_,UJg^7pw*;[q(3Kn,|76 ;bnW;k0BW}yvwuI;oqCXaUj~`e ][3F0[v'Cǃb~ me 5_:ֲ.4|MĞK h[c؈MѶ!|F82aO6xA{:8Ug|m[:d8haa-d@g +}5nmY6ES.mچI=/l+ڋci!> +endobj +560 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 797.673057 101.433071 778.923057 ] +/BS << +/W 0 +>> +/Dest (cb56-1) +>> +endobj +561 0 obj +<< +/Filter /FlateDecode +/Length 4950 +>> +stream +x]]q}_ 0p $Fa cn<%%VnwӪGxxX(*_rzJ)ஷ/\¿buQoW\!$qٛo7\o?v'I>m %,bzAlzö^X,{6mH C1)31Q`4*ADfC3(hkV2]%31hϗ?߁O'_\_0bEvï3=}KAfy," i^"?5C;6W*,m?jKmQO[d02gz r>si^X_;V9]#?5xG;\Cߝ sw9:Gldf0j;[od\utH9z.g~G̨ul4/da(L$zOy6v[êόJfl[z<Ⰺ6S棈gv.#}a#[wA);O\CqS=Rr€ւL7Ǖq%[=? ŨMF=tH=kvިkud( ~#=q]4݆^}'<c\+g7g{l{\XԲۮrաUa=h,umȘ~.Tgu;z(mFKoWtzõMXrm9zz|s[N[ +v`ZޱC\̣^ ʙg<2A~&Fqs,6L٣Ϥm E]:pΖIwxBy/78ÚK,9y:P\䤨= bN`aٽlл(2==TZFE+yz̡͓ydAڸ/z}3[􍋼:5 $N]nÍ"BQY$t#3LJSZ>FZ+bN>IPͦd0׎6!/t| ãGHAj#\gCjև|}pm||rxD#Eu$-+ص>%Oy=A}tqT۴  dtȴ5rn󼟟>Y{G +{缙wssy3ޱ>Y;. ҿ6ۺIa݉!3O9w'|< +3g)o Iݯ3Caw|d#}dgCVwLCM/<~Y Meѣw A5Mrul"^WC6] З#OG7՛YCCKn[YL]V?(pMrc7r~#~ӸtRݙzÜٳa҆Mr mX4iC_Yp n6MM8V_wr/zZAUŒ?_tvJ*iWEL bѦ;b.U+vRn 81Z>T84ޟTMd!rSդ1}Ar)tS {w*&[buqMm ]LӬkZYw oTkG(7עwj2ˌMk! ]ЙMzS; SkrM'n1Ro=[ԳnJ::ۗؑH_ƈZLZ_pc6Ż6⣐wPB -+xҘ04ITᛔ|ЮAۯz%46x$.ubOܥclI t[汕cI5XuC]U4BU@u]VnoCիVY šq^m2WLlnSj] a .4Vu6v*!Y>A&͸_/e3:oQ/kkyܨqJg2ΙtRLASm0qXW4y׮gWkJT7-C&6 F[ ^lMK[G'*\2uS+Jڧthaq %0wVf y%d]a.v]4ˋ +!`yye`Wbr#ڛK[HOr֗mαVC,ƞmAN=p Q&M=ےJ]*E Tc),٪[Jb.+ҜyMZ*x-Z-αN-L,MzJ[eۺݺ:^Y3\7$Tl~]3vT_h-ݧRG_]b[#9+Խ^cr֪[eޠ r&=a+Ӝ*eC]FZ*ҷY +z}YPTZ~vnI?iEL +endstream +endobj +562 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 561 0 R +/Resources 4 0 R +/Annots [ 563 0 R ] +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +563 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 348.873057 101.433071 330.123057 ] +/BS << +/W 0 +>> +/Dest (cb59-1) +>> +endobj +564 0 obj +<< +/Filter /FlateDecode +/Length 4793 +>> +stream +x][8~_H|)$HaA!H }>c\!#1G=ݩb\8>+}_$=ĘO'5oL1;Y+ +Gw1 lBr& uҐB'c'qAP8P-H8C)J\ lH\ l q Cv%.CIG7 I\qכQd֛X 6YכADEuNS-8 M$ yכQXo£C17.ft]x^` VZ+ de+(F anC +܋*1k$1h3^`ZR+1l'CC>ɄaA뚣% ɮL1RV41500 }ks&l! |$O6"f-NAl$|cf9 nfkI"wfyGGm=d2;r+ @@!3%dӏO~鏀ѧ_~_~;GI7!7}<ӱB9嘕W~oz_qfwh=] t,c,?bvvtD \uH.neSO;z8Yjg7CH\8r~)=VMSq}^Y[}ȝy+ m>'%7]G5- !Xqڙpވ>SA=q˒L:ܦ(?je^S9nӗ<7P֏6,vwE3g7HfVV)$Eǘ3#yW#4Č(}y r廙_V=GUQ>]hG[}w?7EݻwwpkՅ:&pn&|Yr$;L}K0f2壖5Ӿ\ Zl)+{/~y/96p Őpނ=;v>݋_3Ed:ʻ n#pQ=#o__lܙp c7=W>rھʸs8oﴟ^B]ۏ{_/ +\Czֽٕs8nyG!e4; ~?] }sׇk\SW {S]y%3rY_c;>|G#O& {~}^ԧb;I글v?Kq]ퟗ83:FsUuw~+$~q\ޙpHU_߱eOaî3м>Z} +񅫶YZ/2yfr{Fs8}ه{"U+\}B./{X@ڙpH~.U"=bgO͸~x<f}w&.< {s4 Խ=G]! +B"~;w܈6M#g#}^ (}`_ݙphx@E&ֳ߫7ׄٳ5%.,eu\jٕכrx"hCWbe4uۼ31@wgaz_lgAׯbQ7#Zffv)o_kKHWgykQH=C#Vg~2~X_[ݗp݈ +=XuKZ +M;cv9bحc;.X忛i;+* +$}Hy}9<!3LJ!''ϿW}x:rSj}efi&7g.[qmqIN믟XN,TrL +W|7p\9n;A೧!wHN[ncB?T۔U~b~U`J?t!7QM9hS~- bgSղHUv!)+o|؄Odo$"U<׊!rz}+g=ۃ`#_Mq a+xV`˸)V1*V]*t 4֝k ^&J;KKQu2ˌJUJ!TI]NU&tfճ^o~ +ctDn*՞ +5UlYSifJF4ס1l9Ud17QMvrAHW(WDž[Q54i~ffT43CST~];jYFr_YZZ\:Y^:*!}&ݚLʚC*e1늺*Vڧr%bkbW67;]{kwWmb_Cq_dX2eبgRIUgV6@ +MhmZi| 2iƽ~9cI^>i~?NtCyu7:,=緺a6Lh~Һ:ZvՈԕAurkmĦBhZƭ/]+VGپiЖC56BX~:8Mg۔|fE('mS:dXFBq*LͲ~]as]쀹6i]dY$p-?敦=}2ٗګKei%> +endobj +566 0 obj +<< +/Filter /FlateDecode +/Length 5021 +>> +stream +x]]$}_Qφ װ6xa?LL,}aPHeTfVO{(ve>BW?h/9=cpϗ_.j^ῒbPOW\!$qٛoox.yL9_IS VPy?h|TAOhbΒ +f۪"p9yI31IP%a]v0D$GAF)QCs:IgR6YEt4)Dty +!$lYiIa)I[MQbU;Y%t%Q[TPZe;u:bO*IM+0I! +:[Yot8t ocJh54ZybLt%0$ +$01N.'Ivr%LʓJ4`\Q4tFKD`l.HL`5lq$. l ,֩%.[tpaqm +N kX.{uF ڵ. ,x:$.p]kSXdu8*J\FF ɤ$.p]wM 1h%.p]ik (Gxp`%.| ,wu'iդSxN>Q'qa˓!K\ L'̨F3jK\8L&]Q'SwAKF“J`Q'IK\2P x{ wVUa-6 @a+i2` +IU=,J,qAҴ)qA$0I8! +6c dB>Q&<{ DlqI\z6' - -$.@D0e[\E\8؂HQ0ٔsDW%]0SkQQ݄XK\Ԕ0%.O.I\3Nₘa dK: [dK\H:%.H  UfP`ZtN:3I\\(T+M0- ?`&9X$.Hr3I\l1H\lz3J\lA'qAVa0Y8tJI\uz8 0.i$.Va%0n ތd[V=7^/"YԇYp`&Qq&[{6o?] X#&]?|{^/ GпY>,S\ +IwX~WWʥ9-r/~*EF]~Y]򅭜';ܽ.s}^ʥ|~)e?r*TmCoZ1涘wκ4ʭT/Byn? K o!um[ӗv,C9˶`)>*{prRVص#{vޖھJ6ڋ ʁ߄ RgWո]|RmKZ +q".O* +K~N/J~* /}X\l)ϵL{}72߽j6{>2jN)wyҙQϔ<]ƚgknq5tck<#qvl6U=vվ|| F0V_}vr9(>uic nW+ԑ{ih7b}5=1G"s qGzOr_ _⑏h3*W,۹{Yw䞏`͝uO%}q@[%s~T@xhw-(Xw2SN6rU&v5Q[21 L7DO?})F5\ֲtb2q֐c.K=%qDc3VxB(KH(ў>L٣Ϥ4|X DLy`x}]-PL*>cag;ΖmOz QnQn7p5y FYrc/+:@ m%JOD[ 9 :Z-Z콅6ЧCLUCed!=U%0?)];[ sY{bȡ0tnq浏FF]MWʚuXPک`ݯu*vM溴NjK99ٖl]ZɄ;N!\'Ap7=7$UH%2"u쭚!tpnU6[[ioԵZ8z6{k um}ai?2[/nk[VXq,tXl450囓=BCǡɆ8=Xl1Yxq,K#c9y:,]^3Q_5~ Jɱoy3cdČ0ۼ(Ʉ\4F3[/hh}Ȱq< +]<aH^ Ft'<~t#Ga=ɱ#Sg"[>{C\`H{`blϯUŝu_[E6Fw[m6=5r)½5Ϻ\7#Q}.ϓ\qrgQG<^f;/{7mg+;3OݶQtQPv2ϓMz|^v*UOυeSuĬ ^+iHhwõ}d6ia/czS}?TGjňLY%Csyqx22g٪e8Zy~%nHWBL" ѕ={kU[slo.1)5rsy `?[cÇ8z% o|uчT捛y'ύZdp=*i_Cˉޗ~{O(ǷGx{=/,;NKs9]N;P{*-|&?Tۋ#1 qTHƈZLwVlwm$vG!\!oIut)EJ' * <3U&%)kKxSf塱<]it _:)!=nK<21y<"Fknkbƕ\}Jmz* ݷr84|z\ӍV.|F1A?Jب6ʘRNoE\Gkc'2iƽ]~)ӜНFje"_wcvǍjla|&I'2U؆ hui}ECǍa|zvՉԍAur>dbӠke;|k {~qtGe\oS>%_Yx}JP< +sle&W\O;b,GuO*+`I[ vWWF{x}s/&:HڨI߸5d-a[7{eس-Cn1ʤg[x^UuhtVb6%[uKiq[LѥW=IKEK@9֩ŒII]Oil[[]g+k놄oR3vKT_ ѩN:Zc87DyJ;"uׄ*jEل7(U|QΖؤ lu^)^aI iH_g)M=@QiR>).;B +endstream +endobj +567 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 566 0 R +/Resources 4 0 R +/Annots [ 568 0 R ] +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +568 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 597.273057 101.433071 578.523057 ] +/BS << +/W 0 +>> +/Dest (cb62-1) +>> +endobj +569 0 obj +<< +/Filter /FlateDecode +/Length 4809 +>> +stream +x]M丑ȳ6| 0|a,0}?R"B셻P] +! +?iIO1|IM)+%VE~;kEar:?e SH~==WYg3v~*S!('TJ[Aq^Py=)I*ڪ̠dLTfX +nU fIR \TZmu0DY#贛<%ѓ$QH:k'H;-wQlqIiI`DJ1H:[Jtx' [(`JKd'"KЩVg\VI:'lZQAAK8#ΆjjI`1DlK `181P^UL(p/V 7Ĥ2z1keJĴ" ''CCd´ Dmыٝbs~; o{zUɔ&W&ao[_j`%r>hzګ_z9~/Z/upN5E\i?+h! ,}Ǫ_^ ZaCQ]>BW^p0 0|y ͙e\c47`G\6nX5e^FW&cIeV}체Qυ[VO9bŖJN^>2,_<*{=l4~Ef#K,kYl,ζrw0ֱ:!SF=#kݷ>U=`!lAA[:VPu=yͣa|̨OJy~)<REH v\u/[=ۻnrYO]&ڶs/5y Zޗu,fJ*cu7⯯bHs;Z[!r$3G{M'Vίv9,̤H=Vz#vh""ibL}{)DJ+:1Ϊ;mԒ:3&2ٽu FA[=-|f X(AoC pDتړpAj}ˬև1*|t~h;phxdˍ2;_mi2Ȳocwߞ8X{ +ֿ:Búc=觱 Υ;r<rX2IA.X3g9E5"JF>h_,~o3InxIk֥m~;spH> cexD# 7v8qX#:r,_vew;#=X[/{$śQqy84_q߻W:G:rv +SrϘz͝Vyf4{/6e;<܍8#Ͻ%Su҅-_$dqpH~_DG +5@lbaE}Ƿ^b{áƽh NqSs #GB8U6ߊ[ѦeЏl2twJ︯`C +W3< !c.u@REkA&_;-_9,FU4-@:TAr asn#?a5&7ޑ=Xz=vm#F^Myպd |_.W]3B5{YߠcAMR;oqVl<5wp5fV$ϐT=?](3/p(͸ٚi\ 08.7'e.>V|=Ƶ~vsM!GvƶDžK-.Gz86WJ\v[w4"Yރu 7؝~ <Ψzc[<63KLnq޻TJj43[P/qE~cu7r~W~vK +MomF^.v+rLy$Fb'K>?8$ʏ~&}=S Mӵk-lgО;u}k|)㋴zFVvt,kWyāQsbuv]le9\w_UjTwmR1/]>:[^XoC򃣷4ҖjG{vޗC>g`[~Wd|.xNsܿiǨU0v_yAfkfLu++"KKߟ6U?S/vmpz ݾ>]eW|>5R(x +Թ&W}fFT{;Kw"/}WUSfviSz_d?a)w6~ƪY By =ԣ/kי]#j:9+>^+gz9bo#fkjౖ8=sՏUlx3λw oEZ?,O ;mSbl"76+g d~͍gl)dX)+e +Zf?CMa7SLDg4f!6reY+h_gE!iQa3l?|y.OWޜ=My +霟%]IN򲠟[A*R ?Nz~;X\'D$k|Y8zůD=H!jmn1o] SKZWcGjor}Wg𒪅ly +|bA^oŹjҋ8/'Nv9 7*&t[bbC\绰]ײ&&pZ;J_Ei~KGmzר&̨ԤlBm$Si`6=Y+K rKkThb)66 <06G?icPl6PuPB -kx(f'EgpM| ޏIlⵓ7!B/ܥChQ 3c+)sT$c- uM)q%jNŮAu%v6Uz2]k}W.mu~t0_(Џ$]lT[ɘRNE\gkcTcdҊ{9T9cIoQ>8:nTc Ӹu%+Y~oY<(16m.ֱu m75{-B}ĦAh</[ ޱ |hkp_cپ$Y8E-VQn H~l?\쀅ve7]eB`xäI軧׻}1ڛK[Dz,ЯۚcN֖es϶ OZcIcpTeօdZj-QRh1E{QiNt|Tlnx-Z-rΡN-MJi}ڞY7{dz5)y3w:CA*UJ&D,Ԥ5UE!_(ț\ϕl3"5WτZt~l+鶂5U+WKJ :ͩM&ʱ\- vV17+**}R~Qye +endstream +endobj +570 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 569 0 R +/Resources 4 0 R +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +571 0 obj +<< +/Filter /FlateDecode +/Length 4937 +>> +stream +x][q~_Vxր <ٱX!?_EJ= LcUI-wEC)x~|QCp +rAV]08|tןX`3_/;l_u )y?b VP97`T^8Ŕ$p -UfPNR 2& +3,m3L$`N.*`6J:t %у$QH:kH;-wQlqIiI`DJ1H:[JtNt]Q啖t.E"QS1ĥ(7htZش+ +䃠3+DlqVIga Gҁ' 1 +jI`1DlK `D `/H\# )/qO +:|8ve:B$.c=$k #-p-%. 1I\G ơSJ₤#MC`]r%.HV!D'd=ǛADEuNS-8 &IybI\x3J\7&/jE1Q֋>Z-"๵‹-k3B/N!=Z^_.{_h XOǿ7חdBJe~_/K>O[9+hyn*Om\8`-<`EbtzzðШk (|Zti>@VIsفuLO&ʭrw +ΏTeƱ:5]M:+{tH TQ&>ز} +sm8een}rԭ86FjV])nMy]6_m(7ةX]nc +Vlϩl}[aն8?j>׶Z+64d``s!G|I<#"}j4;=R*u*R̈́ D1zVP[WO7& ڔ)cck3uns[fX6q,s:,&f^=:&Ԕ17X}е#;6ZO`n]+ oXvO:òvܛ2#x/d?7[5cF}b~/ ӵ׹7ڍQשּׂe O.߃,YmcXax;n=G.lzxud/s976|.x\ p+úg8]uN@Yv @ڍ7܏E+u;ʍ75e_u&֬حnj&Ĩ(;V$(NHLVye0+#2`2l2[}糖!6.gκdVisߎTkH%F:gC*M>.k;Zr&׬22v+vn{ۗ]nl˼c}6ctEi5 b!H292iC(;v-H 3[WP8~̠yF6 +9 +ufZ#R`IHODʵ,jR;*u[g<)\SZ``!oLJgc{53V>KG#m۟#?ѝWZD:gC+"]D^m-kblӚ7:̂O[= 飑ܚnTwGcGo<*fik1:C-i-]кV9ޅ?AHsEhCey׋z|s,ץmG>n]+֠t&9@VY<3ݠ=qWZlZ.}iO;'ڟrYAmooXӡv'QaU=qot{,ݝ^b=}H/ٹ"ꟸvw=xݨz}em;1*J ~ W~w3=bhi{L6Z&wy aOU2Zڝpr3߅Z:Gh[wPJZ.V;ʋEڻB|y鐷szfWݵ#jhxx-[#:m WQ[Ap_8hK>Vtɟ~_:Vt񰞝fcʧ˷j;/3v1#zȃuNƯ}>f:gCUڹk;͸vD`!ekW|;eoH^]{-F\MNum]9)'mM/u^uNHf{r~/GfGPN߷:XtHɆ=gTM򉙷__֬"ッ9m߽jC\a[duYg^u6$1+;r)C,aSJ֪~הr-:ry+Sp9> +endobj +573 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 770.673057 101.433071 751.923057 ] +/BS << +/W 0 +>> +/Dest (cb65-1) +>> +endobj +574 0 obj +<< +/Filter /FlateDecode +/Length 4841 +>> +stream +x][,~_1wt@08`8!sap򐿟-Z]=;eLWR]$hYKSJ9wpZ8yCH`71_No_Nu}6Shwrv)FUJi+b4>'bΒ +f۪"p9yI31IP%a]v0D$GAF)QCs:IgR6YEt4)Dty +!$lYiIa)I[MQbU;Y%t%Q[TPZe;u:bO*IM+УH! +:[Yot8I:1%VY-Q'q˓!K\ L'F#jK\8L&]Q'SwAKF“J`Q'IK\2P x{ wVUa-6 @n+i2` +IU-,J,qAҰ)qA$0I8! +6c dB>Q&\{ DlqI\z6' - -$.@D0e[E8؂HQ0ٔsD0W%]0SkQQ݄XK\Ԕ%.O.I\3Nₘa dK: [dK\H:%.H  YfP`\tN<3I\\(T3M0- >4`&9X$.Ht3I\l1H\l|3J\lA'qAVa0Y8tJI\u|8 /i$.Va)o 4ߌd[Z}%"<N_ßϧo_}.uXri=Qj~DF}4ί硌AyÐ2CaײZ(^g?.Ne[͠9%Xs75M,OKY",] 2ti_j"- bT݃y \'ߛ!5=ԺþO. $ڻFz&=s逬V"G$|e\ӓñ;e)WY%C={=}[3 4mP߇_lE#/48ͭY<ckX}=:"|й\9ȭAy,9-Q½Xk2 P{bdwy4G!v^_uR4~iJoصa+qa> qwij<boij{:OA<nĩ<+̔.N'=H<&=8:ѐl߶D{0NocxgO3j c/3<1YKCN]l9̊*NE-;'Spl<!TO?1k*Z 2W bT&]rRy8^ks ~#~̽=&7n5؃y zZy2gu:!Š ]q: jBYT qVl>y$45X=|=~k\QZ&GjsWGQq! iwVx zAy8$_q0 =d{l{\XԲ;cu!Fmw>Kvw42X=0}cwiL/~ <Ψz[<2%{{ɛ3ۑQhg$L4P/qD)׳gc5}_Ϸp,ϣX,vmOtԑ&v[WrFE5j Bh[\VTZ⯿7#[ӵEԽaOzbzwF/_eSu~Zgz8.v:,ݥP{!'o̺l~UhM,\fd谘-NʖNlZl/Tv.Oږ,}ٖS쯾 ^AۡؽU_ۂaɛnT9|3W 8ͯU_iJlcx$?[IGG닶foSZDD/[j s#Vä2W3TX/R%BVnN!Ô=L/mᗽ!SMt>T[7r p/.D>ovX~wɠ%r>/p19))=Ā=U@cm@e:yjң_^)ΚCydָ/uGЛ 3h 0~ }!fuxhkn=@A[v^Ir)k{p`GuqyZH^qu=qH?,[fק=O?~do:b>x0σ5љkK1QOV*<@XlYo\}$~1<RU>پ?~kgI|HjOZz`vl> +endobj +576 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 270.123057 101.433071 251.373057 ] +/BS << +/W 0 +>> +/Dest (cb68-1) +>> +endobj +577 0 obj +<< +/Filter /FlateDecode +/Length 4725 +>> +stream +x][,~_Qφu `16xYbOʹc?e.UigÞNUF)}Tfw +R"]o?_~HĿ!`PW% s/|ڻum˟u4j^SR>x/-PIe{mrJcT0z4WyA1XN3H0䕙"^*ZϨJK`$2)+q: QGVS^s:ctQ8lt[lYbtLduEo8-]-@W`uE:8]4@"t20ii HNauQ$-N`5Rs:[GA"{AQG:A:DAt\ӊ##a%1Q0rYl18.0 M%%|KR9 fJ GP.9.0\t1=kЮ8.0]CJr\@]c0* kxHs\^X:qwuqw4xW[7swjҎE$ wA!w` iq +n "?fL +q\`*X]ʣřqVʹqb2X.3p\Q5 fjNU:J ,x~p +߄.B8.]ВL c!r\#+18$=9._-3 ++zӁwM&'@/Xs\E \>V-^B`AY' Xhbkbkx"*x.` ~w_`<;ÅY-q*eyvXn֍~35ݪRW~OF:N7Sv!CtW|!C7E3EtW3.N t=+ȮE 4r]hHYBnA8&ӥXա*&#/3ܼMhޜ'6N#K/ qs::.B4ݗ!7s_=uđ icyب N5 )jd\~7Htgn4W6]R!R$9ޙ?ApsAȠF1ΟYun[cѺI*w9;CVnYgYLݻ+#wg7CHC28s~=Mcqh,|ߟצ-{~lȝx+mDmPۊ8s:oO)Hrr} M?Q0|z:cGM<}se(sZ;qwGW|ۺurڴ/:҂F[2'<Ꮌa㫽ڼ__֣}ա>vw\$Vy>sp>)mV ȝyNᱞM>ϣ+yΆ*k;{fQ} 3鐲q+~3xav/jO\vMWW^K*+&wX`ێ+'9^Vkޙth>p ͐pto܁=C|O"#kfHR{9R7aI1f<";w9"ݾ;Yڍ|GZۧ. h_]9}$?S/׹w9R7|w̭hNvݗYpn霾 +q;><'s; a}v5\Nfa=tW}A>#=gݙt;OsG"U+\}B.o{X@OڙtH~.)'O͸1~xm=L\@ob3{ BOݫQsT+尌6PQo~p k8f6~lyΆҸrOvwN0>𘀭g s:;; خ{>O=k`w9Z70d=PTrW3S{oGmW?5醽}D(u3tb |m;9ꮟBdw9#OHw|$x^r+;=E!d%?59-R]}"h>\7ׂxN mryHf _>_qM=݈CDpcJѧw$ovUJ݃I Pc󻱙).#5Q* Jw RZw erB&t+OIvg(6`9̺헋IEWG"RARа꺐sx}/%Fe^^urs0e/*H퟈S9UvIo%FLAwp)KYr"}JcRlR~̉^{Ō9KFqzpf#SK=Z7z&],XoԶrz4q/Un̲*l&~(*J￫G46xH!i.}l^*H&}"ݒD4Y#1HT +8QXn•Pf#P]巉iPseN5p*v;Z/= Yx&6 !WRZٮVa *46h-4<~Y7. Ӝ6VGpFw^ i^2EmiJiDMxK8@g'kKFEJ5ɲy4;϶4g*k6;K=2vy* S1QFkLNpBB9g=V2XI[C1+m7(ⶢ)m|,-J5 +4'g #BЧ+e KņgId?q +endstream +endobj +578 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 577 0 R +/Resources 4 0 R +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +579 0 obj +<< +/Filter /FlateDecode +/Length 4962 +>> +stream +x][q~_Vx@ &D]N#鬝| KEIG%%-N)I[  3wL-$%qop qI+@WԥI%.p]В"Dԅ @'?T$.]嬕wuF#q0l1 guRL!J%xW$.]6%.&J\'xFiJ!x |Ru~µ ((K +$q)%.Xc$.e4o1<-l(qlH<1ft-"t:H\MA/q~bwI& B-*`*g.4l-q<]贖 oPh"L$. XM-Hq%.'v%.0A u4!qm'qsDl$.p%.e$.69 $[I\M4R$o(qA +&BStm QC%QG5n.uة':n(؝:_u>-}@4ߚuh/뻁{w(Kv>@<_mv Y瞧G-` d^h _uyrڂr_y:J݃^XU]m Bu=Q/;O4o711Ok=uO^,Yv[FkE;1;78m[6!;۷f#Vo0嗴VWԓeTj7:za+?۹Ww6K5/4DKvi#ۍ8mZx3o[NҞ>͞,ۡڽp{7lM]qPV_ [&Vw֘{e{( fȘ*i^;)Λ/:tnCmv=mS1; B)s[.긐ۊy~T}*nM{I_Crl׽6jf=ok[FxrT'Ǟmʶ+X/e+w9S~< +?ѡpސ`(ѡ =˝Vo (@bُ%B(7żhQngR/ע-Ha!ͷsS򙺴/ʜiQ'Qv.ϓ-lA^b\v!f[q+u%_> vâS__N#M Ww^וTwH=鐚vPB.GvЩ-grͰ⮡݀ b\/QQFֲ%2͒'yG[}lCWZ"y2ϳ!]8=PYPg Ku4u] wP|? 3bQx.ӡ(F8[1%'"kɞKnw2^A5 j3"}o'):!}>wC=9>}0Lvzha;_92C a?jG3D>y:4duOBT|,;{+d]Y+^ᯞlȌ'ڟ?7|6jQ~:?{vsy >aݽwFSWV}Ho?ػ?zzrߨz'^QQw72wP~AƀPe '(ǧGzw|dýOsRrNQ{.ӡOU2ZU% +WW> )ʱ㞽T[=1 8vHz/Oy6䥃}q =gnaW /C%~<O?SA#떫~M m6 }`>|<}e(9zV4ukg#y}ͣv;/|,`KޚZE\ \Rh| GSC_8m^uUX&9Qy%i{0UQZB|WxK,#a#My)?/yQ+ۨ8׈_9-pZvD|܁_^W;{Va~헋'=|گr 8 ȧDZҥtnX}3/5 +Zc6s7K_r|щ&n|B>]eq4墉. bvhÁWU˂ m 4R/h_+w]g*l>oX"[1߽"]|5_V i[-X[ VPG󹳵|V])j(}|zy)QUf;Rv~D A Y]0 QMJ:At\PSŖ5o4X^ݺT0J7F-_b>#nFx8#..] + +zʭ^oE'hgpU̓hSpUf:$vͥC I-vMC*1IUXuE]3\ U15P~[Uje^wpw&;zq=Hjo-|FK%MlTZeRhFoY\Fke' 2iƽ]zӜj> ]Pq*[ʭ+q?ēb}"æhuli]ACǍZvՈԕAuBڰ +]m2nGxZ:26aϗg‡ p.c6%_"qI۔-.#.BqUe+3}%~r]EFm:jl_tE_0ҴG0`i*}XZ O|חuα(2x[ ղl7lurh.zvز- \_)٪2QTLY-14K(R>EoiJmDߤrˌW%[ 9S-e% }*ٶnZ]fkw 3U-ۤ|gyU)UWBtlc&wTB&b4pHY+kBwLd +P6 t*~(gKJ@X +j46+A'c^~_F!|&j$?2 +endstream +endobj +580 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 579 0 R +/Resources 4 0 R +/Annots [ 581 0 R ] +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +581 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 570.123057 101.433071 551.373057 ] +/BS << +/W 0 +>> +/Dest (cb71-1) +>> +endobj +582 0 obj +<< +/Filter /FlateDecode +/Length 4847 +>> +stream +x]ێ$9}gM|Hi  V)`?;R9;*CTO1U>:㶪3\N^R gLT fXݪ20Q;I3tQPi;QJ5N)A=Mu:v +i%]B>I:[|VZysJgSt`-&E+"l1Nt ]ItN>'D]N:䓵Ji=y`ӊ:(tR|u[e$-?">VI1P^U2P^VQ3P _Z#0+]LPj%n(/1yoQ!bL1,Ny_;#&}zY)?BI_a5oQ)3YCF܍ԠeӍG+a8X aDtE1nyn!se[xWwy/[mLIKHi  wfyG (,:c(#^WnWN Ŋyo]2hߟ~??O^OB)g~_ta]_y3), Uבz,!z\w hj=|#V}dm)_X>3> wPz9,ny:ވ]oCC=ZƨDIӅM;h|``CtKeM]; eV_-el_d?k{<r8bqnB>)Ockk={,~m;IQ.ؾ'sֈ(;G:(/lD;I,oN CMutH~bE8sڗ9?'97CV8ck{--vFm+>t}8#]-ތZd4b܃y z)~s#wmW=\S*{gM<#y}t/=;UWйY)]X;>yLzpu0!\H~ms98|?ly0A/Oޱ={бU$ʸҩ{ #GB8U2E-|Fi[6R{KFโy s{;!.3 +tX bBtR]|? sym9Pt鶜P|sŎ]XO< ڠ Y({f/s331]lIϵd=mib{^k5+û阢?٤Y.蛆!lxmkE^:jnkl +moz?r[~3_> v~$Қ<$ٰAfij7'oGmygo,6SV悑RcƏO?c?z\Wnl,,Cq<obApo7ҰTz5hٔ=wn9L*Cj3A-`Q(ѿR98ў[}GIc-7=d/(;[κs< ׅ(Vn7p5y FYr WhlY bg*X{63QA&͸Oe3ZLq}X>i~y7iܺ^ޔeNU؆ hui}ECǍa|zvՉԍAur>dbӠkeW-Ծ|ikhqL'WpuOgV$(iҡue/-[I4דw #v,/J+X_=OPGI7,2/ۜc˺Vec϶ Z(mIyV։dZl-IRl1E{^iNW&-[g-y[zK&Ax&u=mn]wmϬzdz6I=2v<î0Q;%*q;H+5;&g^g*vv]:e%6>[Af'Ϯ> +endobj +584 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 142.623057 101.433071 123.873057 ] +/BS << +/W 0 +>> +/Dest (cb74-1) +>> +endobj +585 0 obj +<< +/Filter /FlateDecode +/Length 4736 +>> +stream +x]ێ丑}ϯg#~f60@/ᇮ)5xeR %ʫw#wzE +S,Fx{肽Bva\#j)iTF>x/,P :FKS 2k&˩`Q*03RUf(/ 2ZϨR `0F+F'P)9Udu:zpH+9]s6p:[lYb1th-*xuJrfuQh$-N`Bq: [ 8xj[n-t( -h|-;Ѓ%p$L0c` hx@ h[qh [ z.-: -HqְEq 5"J 48. \)8.`\@rqjp'9.`]m@v  F + xWkdu8h4.:`eP! xWa8.0]1:?j xWY\7rwUF9 #Xo +;[@`Ni pA-wei CtX h>VQLE-ҹqbX.3p\0* ,xWjԜ+UX.q0H8`|tY]D  -) 0 +"IX$qxC9.p] Vk xW=8._-ӧB)8xWH0 +#K&Γ@ `/Xq\tpXpl9. buxq!8. +h8. +e%ovz[v~`` 3v?rXzgkÇ; u(c;09_S:ʗu/Uz9餞5,[R'˛mSNҏyS4:5]9/oOz?OM6rA70MT@\:_,Cb,J!(kH5֌WlSS/K.`-ނ^1fEbݮҙ百5wHƐj+Tk6s:[/?7i߳\Wϗ;Vi_fLΘnAۇmu|Lߜק7PfFDY_ 7{eu.33 ۾TjKi%̈aske$F5BO nKYEyjܣm9Oöt;kYcVLo|@jT#/ِ.rᕤk= jZ6CƼ} _BI%?)淚č'+?Ɩ>F˟B&KhOٙtԻzjjdĭk}}Ow,Y e4o~ƪA`NGa_ӡ_-c*$lqz^ԖXd-Ys2FWe?Lo<m]d[u>Aư>y>dے~Y+U-6~s.-xٟuߟ3)C%{z+Qs62;eYh9"z#/hcJ-OMygu;g=bs6;>[D8#ѝz6w9ޕ?zApsFNϨC_md/k +Gw[I!Bhs64}?v~a͹2̧U8nݙZ\YX![?y:"kc%3y௾.+ټ VQZ`ٳT3P{O}7z{UѵcI\\gb=}p޹o {yNЃk}vuҊyn(NwWZ,!'djVm,u|/#ۮZuhWqD;4v}yΆZ{ +-H~:>s(k +&sl{ޭTlbYO41]3l{R+\hju \+# EiƠH2s:ʓfI;}򙒿.+?eQgm l>oՙtH==g&y۷5@+; IV#sG#{ ѽvd1Ti|{1Һ3k='$&ˣӽN?sٕ=f~xal^ܸ. RўWgZb㕶P BI+9ǝBͩC'F{L;Vs_"OQ˶ mh1p)L v]bRA?c>9|loS6ڀÈH[PL3Btr-ŴYf|)%!,'4ur’P,v-UqELGJ\f몓UW͹']PnoKJDʨ6 br;1ۅ~'ҩ 3V&aH@29]L¦fsK*1|1@]\lYiߪ~.ŎZ nJxX{b]gR&B]7(x)RZ8BL2&f)1vRQ)f/+:i|I[U>c/H75VUV}LIeXeF]\Y E2P~[6wU=dw[.s[-%ە0q`3R!EDJaR(Bo8NqY:7.?_aNi"~8-i˒G?sShǕl2.HFN1mbTBe*/bJZАq*6%(ĞR̠2kk-uXU#5gۦ ӡNb4형A}98 Eː|N#JZtIo2>)cla&.$W=oTVuPtΔa>uWQ% Fօƅ=w}6٥@LYVK4/[1G_ uRC\{:;9k,jVP l]IEitH|FlrsT-|)约m*m*nΛF-ξ\()/sg))u6˲h{.C\LEfW)GΔ:p HFrwsCzyc%H%9x0 xN6og* HиBTq]6e[OƿDq +endstream +endobj +586 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 585 0 R +/Resources 4 0 R +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +587 0 obj +<< +/Filter /FlateDecode +/Length 4893 +>> +stream +x]]$}_Qφ,aw1?Lܶ1}aO(SR22+P}BGG!2K_~/9=cpO/j^_I(?7V.N^'8M ׷ .y˟~uu}5Sh珤rv)FUJi+b4>'41gI3|tmUfΘ$̰.jUeav +f죠 +vh!9kSӃ3z +),"t:Ig^K<|tk甤-Φ(l*Q[LVEb,l-*(-鲝|NΉ J1muF'kzuQ-A,`H: [~$xۘ0Z nt-|L-*0&%LI$`\rIbb 'rx* X[ X [F-6; -H q8+q:E u*k ,)$. \$.`\r,qZp%.`]냑vKA ޵N+ xZu8Yk8.:A`e2)) x$$.p]18a Z xx4A+#qgQb4x9- X 0Cg(l2K$.]1 cl5锃OIGx\Cx.&*+q> 3wL-d%q"nF qI+@WI$.]ВD @'8T%.]孕 wC#`MF5a +c BҁwFK:R1K\4mJ\" LN!)$.YAOԅ ^₨ ,Q[\t Ih`5Fh` B LY#p RL6UI̔"Z%t F7a*`-5% z @K 贓 f؂"q)DN ҅Nk -%.H&a%SL$& +%.0JL)` hB-X`e-; R2L[2ތd[I\U(L4Rd4NKZo& UXob'l7EMY$7N.9Zo& f`^Hތvxg0"2 MD^x}D|&+8c@seg@F7 `&V)VJ6P_cn ߢ +CĜ"52cX8Dmѓٝd?vgL׿Weh%C|a`2#h Xb,{(1Us7Km62 n?ld,rc%hOZ*X%krDL탖me>b2' >m İhFW4^|V/7ނ ;ƞF֣:t(eO-M| + xDil64Qn7. Ŋy]2iL_?t~z?r?~_/k\{dqH濴\?׿Յ*KnCy]y+z(KRmQ=-ْQhÏCE{ƨGoo/i βZIqfC<5x%`}M>}2Cֱ0j2tT"dR,.{2ϳWCVEVYϼ{}^ֆ^:ɬ(n>p_HeGݺvPj(=P9vqj@ CH *Z 2a|ėlM7$uL=? MF=tH=jv֨Q?(`5[Gb{1i Г+yDZk5J_Gu23zf1Qk^Y ڐ{cyu :XCKEe؈3m#ng?6񽄓y 8~ˏ?qyW/EƟRG{2( J3.dRвY!յj_4|^wly1ֳgZֱ4fhcu!FmCR]1wt;4FugTF-sSfȈa1<~f10|YCSouc{~sywv<4]K$zcǦĶiR{)aqe)쿊;`mo7'z]{9:ԮM*=7;OI;4k7Zg7{4/#!<<@/uF!;.B>K/12'ew6e=Yl'}z:-s Ced^{3[N3w2ϓ,#Sչ#5xFKtSzYRvIO//aDאgd0OI~g)#* 5glȋ>?4-/~|-<;,yHeȍY7/zN#*ï9yd'A;̪P<~e.ϳ0g&ԑDd*H᭗9x_xĦ}[W<)n;1d'<ĐcOFd;#6st=vs c/>jwbDms2ӡN\9^VDгq|vToC+={k%+Z4'<M>Ms?Y;x)W昩%ݏݽh Eo=WMv:Nve16r餄D-uJ\1WyźªWrF. o+qӷU,hwиcqMnrV&67 VFٮWƄ0Bw}+:Z;, f2 =odQ+yܨqJg2tRLSm0qXW4y׮gWkJT7-C&6 F[ ^lMK[GˣUe:ۧ++WO:2KVa$@J~ǻR]삅tii]e, B`/xNJwPGI7,2/ۜcc/uv,{9}h­.8F4lK+uUN. SUJզdn)-Nb)ӞJs6i:hɶs8:SX2 s7)mmvl{epݐ#SճMw{ΜguR՟Si>&j5|J!C}Mtm> +endobj +589 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 438.873057 101.433071 420.123057 ] +/BS << +/W 0 +>> +/Dest (cb77-1) +>> +endobj +590 0 obj +<< +/Filter /FlateDecode +/Length 4866 +>> +stream +x]]ʑ}_Qφ+,kX,fYamw{" eJqizȓ'#C*I~/9=cpϧ_Nj^_):(Y+'sH3 lb& O:ﺎ>)`?'R9;*CTO1UN1gI3|t]"p9yI31IP%a]ZaI*J+ءR(tON)lh$SN{-BI9ҒS8 Dl1)ZIawKJv99'2t*t3O>Y$֓6Ì ( l gQ[UFY#Ɣjhv[eŘDlK `lQ1H4/I`b\6N"K 'hd?Vi茖*]j0sElI\` lA -Y ,-XЭSYK\`N qL%q (e ,8ׂ;%. ]L(qZk] +XuZI\֢-`d%.p]Uw1 8 +III\&)'q%@ sJ\k28q&H\L%F]P=tJ\&D˟0VN9H\:aD]tDžy /O:,qbx.3I\ U"$3Ct(6ӯ?c=_LYJi +d Gq6 -DRd! |s7R?,?m>cDŽ3ru*h0u6͆ C\ܪo?Ŋ9X/vƟ?|0zSʅ|כ/@m2ϘGhu9j;"*xGO>.nKeJڞ_Q';c\Y}#UpO^/G~dc7sӿZD p#cUރ/Yluo@+*uh(]FR ewnl9.e_>jkk2fqJ۵ ~hOe +/Ӷsrɰyγ; 둿t:? ;RU䍈+@Gn~fHN=f5MGD401=tv7s}]y#%FniY`Ha*mjqFA}/붌^ O_A8K:@:JBC߱:OcuIbGx8cs(")Jy:ėGl4ƭƈ0wXRm5Gm%݃umt. ^ݘe7s3c}e1i1ҽǒF[/K/y_eL+/y6 ݰR9ye!GKuk'ztX5aʇX>[vY0]v`cuQ&B##Ko}g;uU6=7ˌډYu )v_,6mGWcnoˮi㎓Eucs=ImRLY%CLoXCN_ n}u3sÕ6Dͅ\ʍypUfRBmtHɆ=gd#i6 4>wX鐺~bE8xuZ½`]+Nֈέj+ׅss; >t}݇?ynԲb܃uͷcn=}jv}ղrfy}G2yփu }a'Nsdnjta}޺6c҃;u˹~H\tԲcL@/O^=Xлso8ZîVT +Vf6ڼ{yǍh2܏l2t?vJ`C}|;z绲^ өoN| U<%`<Roʁ|!6)@`Cص%{5ZU +eo#3D#j]i]ܻ:fj\5%[3K1yHh~J| '{!vzϓ4 I<4JQ]ߕd{n=P|RV_ 08.7'u.>V|;Ƶ~Spm k[Zv꺜c}Z!Tu2>٤Ye kO"@6zm@j rA]k'l +uw|-m LX~=<-=KEv'/g\ IzMgs.GzP 9(3Cz sŝysܯx0=Ї y^#)p5lBy./9Q!M[M|'ct&]5˻~i)z ]`~;XRgP=h{y*dA_N;W1 t[p삀0Ny,i}}47[gC}Fdwsǂ5٫I/bz}Zr)>T! :^7+UL诅uU{4n.ta3NeMdyKx9[;JD)6UkTYfTnR_Kqm$Silz6jY*\+k9At9|J^5WwS:mjdEz$#֏k1$mvܘ+]]vQ(;W(_DžIut)=hYe +ߤ*͉J*KEK@9Sd% v^Vunw]m4\7$TjCܙ YT)P;u*qFK]v:#Rs%' UQӋeޠ +T}\-+s +42A;zKIH[Ez7Grvi>)?fm +endstream +endobj +591 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 590 0 R +/Resources 4 0 R +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +592 0 obj +<< +/Filter /FlateDecode +/Length 4800 +>> +stream +x]ێ8r}ϯFK5a6ᇪ]5f J))j9Bu"H1DF>+_$=Ř/'5/XdY+ +Gw6L!9þ/;׺>)%o;N!JP)msSyAKLIR е* JI*ADAa* 4I* J+ءR("kvG{Igc2It0NK4y]t[\RZ94tl u`%]-]-@WuEy%]KH%T Zg\VI:'lZQAAK8#Άjxv[esŘ@`%0% + $0!L I$`(RX4QVD>-UőjBDX[P@ka!+q"qݒJZ % 3k' ,(PJXpwJ\`AA ֵ ,hRX%$. 𮵨)` ,qq:Z k +xH\QC/q25Q5K 栕k2xא(Gxdw`%.zd["qO`I%.p]1|.M:Džy^IBe%.p]g?aE58X%xW$.p] Nap$.6i芺4 Z2`T0HZJ\Xޓx"=h$._0Nʀ)$xWi*˦>D `/H\#M)/qO +:|O8ve:B$.c=%k #-p-%.M1I\G<^tL1E]N B K1%.O.I\3$qAHSE,ES4%-p%. D[ J\MDXK%)3Į&4&$.p$.@`a-֑171&ǛA₤` <> ơSJ₤#MC`^r%.HV!D'd=ǛADEuNS-8 &IybI\x3J\7&V8Xu{w|aH ºni3DqG+}׀×{sMЗc~vPJ0.<:p\}N斿\/.e,|.2rrs\wu|g{qϱ2/6Teuq~?co ;NnqkKڝ<oKgkg[s!gS9Rv>^3[]>/erkCYmo aݮu>?~V„snlޜcl֬8e,. +Bl2a G:wM=U{l˨O?}y܏?h`c lJ rv~> ^ pFWq{ўØID/=\QsO伽Cm9W饡yw#{d[v-9!AνȟջLҕ"wy!lr둵+{yB dVn +GLʑglXC1 uHY]:rq${.FvE_sd>xO +.!qoo#XCގ9dXZn|ڎ-\񎴲m‡F^ɯ>kooNoXgC+KV^yFZG|ӕ7̂O[=14HnT'#50by:nO~4'z=bUcu>ΧqDwA컁fӡyGw<iu٘}v07#b;m#;oI Jxγo^?O׻yɫ; 3dW/^[\C뒌Lz!s_USPuW3|g~nˍMl{=VPOox hЍw RtΚzگ*_|ܻ?Xw=xߨz +9")g~5 RF !Q\P=>t{8>=/G۾v[IM;|cu aOU2Z2X<Q1~{zvOx +rx_ߩ=2ss!gҠ˃uy;j #;8)]C÷#bɮ~'mi~<w)랼Vk*:(ȿ{Z?ee ׹q=+ш`C1s^;IQ;IY{WGf 6_Q{Ol(;ѨK9D#38* +>-kݍe}|i`݃>\z\p™l ^cIuJQr׮}[[}ٗ ybKg3CW(y:|xXq7Vwqic|:;mzΗYl, zȃuQ;wc#2|G<SǼs~,3jNdy:}򊿚o| #yY%WKWS(O8}öy|϶&{<ZwjHق#;܋~>bgʤ@j:}u;ٰI>1˚@Old(*ˇG׿epmmFثnwˊ";0-\x6cpJ~g4[{/-Ϗm):575heviMܼj->2K{cӲ -(_0Ya$_հryaWqҜ=Mtm0ܱ|f~/=yZeǵVÍ^NLl Bi׸B^hVqT /MfVLJyMR{;jܬu'r_YZٮt}騄LtNR1IZ+ꪘ)O^h]Jtu:U󺃻mb_:֯v/K^h3ci]k6@ +Mh=l +$Vo!0_{Y?(G*{Q-L֕_b7,:m:XV4uܘUWKI]TWm/t;5v=6#ѱi |hδ} +06mK;%h[ҡ%UARv fLdHzVwй|h2۟UtE?1ѴG0q_LeR{za,'Km>úؼkl7lurhzvت-\(٪ѲPTLY=c(6Z|QP}&[ӮډU[V-Ρ]j%/kW)Ȯۦue=wA? v +endstream +endobj +593 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 592 0 R +/Resources 4 0 R +/Annots [ 594 0 R ] +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +594 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 713.673057 101.433071 694.923057 ] +/BS << +/W 0 +>> +/Dest (cb80-1) +>> +endobj +595 0 obj +<< +/Filter /FlateDecode +/Length 4861 +>> +stream +x]]$}_QφIa k?z1C6˝>ʔ[}gl_2BQHWo4KNO)凋WWR.xE~Ov%Ov˯.Σf9TN1ŨP)mSA&,`θ* T0UEm 3LNR }TZmu0D9g~ +pztFO!eE]NG#鬝BtkIOtx휒t%]-6X%`IJ[ ]-@WuE%]9QS)(?dtZOش# +3%(ElVIgaßOxSFmՒcu/1 E QFD$qqr8LK.I,`R\VBX3Z"`wAbauf'q5 $.g%.`[H\`ANe-q: 3kSpXP.%.\ t1ܣk}0XЮu)H\`i%qZ'wVQ02W'0H\L&%%qk"Q'A+q&H37xxe$.p]L@?J/+q{茕Ma3?aL&rwuB< ^tYDe%.]g?aF5x𮶘%xW$.] ^a$.6i芺< Z2`T8HZJ\޳r{DlH\: [aL#XSH:aIU*f xM B$I Q $7!+0K\%`NS9I\ lH\ lA q")kD.Dɦ *邙R]&LZA/q~pIv [0T$.T4%\i؂ [Di-qA2Ѡ$234֒ps֙ID&Xi)ElMH\Ŷ,6uz'qAJIₔ`QF`֛Q€` "> +ơSJ€֛I\qI$qA +M,qmf ;:؂) &Ib @$q 7ы[֛Q. ft]DA\`VZ+degw cn U ȗH2q*Z +xm8a}[Tp3SFf 辭9z2l/?v0 LFw{Dͷ)w%jcfiFfa£퇕0En#KR2+Cdyke[xI"^"#\+y/v~"ԋ=郍mdҡa#>I43ߙ VX#" HcrĿt.V풙Ncr4˯o@_~Fݖ\/y+)ݐʮ:5cISҫie)6mm"wl"yfy_i cXZw-%;B9W?}oOۧ5 J6Qp_HeGݺvP(=Pz"TO0{*Z 2Ww~w,Qvɹ{H=zhlk2)6v=ؓyDZk5J\pg 5 ,mȽP^7XPS~wȢlx߈3bsdpddOy6uqGiy"5ƀ8 +7Ҍ lVH{u-#|Z0$sX۪u/_g:S4c6>.2cx@x'!{{̟k`N`tPi;Gs.F/x޻}~2ϓ,cMHҴ.Y.*KölIHlKHB̆Lǯ >7SCGˀĂ$o͹Ny6y1<=lwünyZy! +. tח=/?qeTC0Oyu>B{:J=fÿ#=(\?rRy?Ry:C>9sL|7HZWx=k;|6K3toGk7э]dƇ9ِé+!~~Bh~\ z>`夳c^rBePആI@ۃz8|kmH ,ҩj/,_^22M,(շn*Y,7m;b7OT&-_M^mmKfj0/biæ~r.'&m'Mt;b$Lt&];9ER-8i:Jt#XP:n">wUԋ.~9N;opZYXCAvy))P~ÊGkk(Ej"[N +VLW^[ iUhXO ULBŪb=m ]LӬkZYw oTkG(kQջF5eF&5eI.Uv̦gFTZ\ EL[OŖ*zҡNe7vS7ҏ1l}*1ܘv.~(+Bm:^4* *SB*|5HIWBcNXI.]:ƆY/Hn[I˘aϗ6с*\2uS+Jڧthaq %0wVf y%d]a.v]4ˋ +!`yye`}19@FMƥa$m'Km 6rVec϶ Z(mIyV։dZl-IRl1E{^iNW&-[g-y[zK&Ax&u=mn]wmzdz6I=2v<îA񘨝*q-%5<{%WkBwLZ5lUQΖؤ9@ +46;Agdj}D?6ݥ>O{Rp +endstream +endobj +596 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 595 0 R +/Resources 4 0 R +/Annots [ 597 0 R ] +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +597 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 273.873057 101.433071 255.123057 ] +/BS << +/W 0 +>> +/Dest (cb83-1) +>> +endobj +598 0 obj +<< +/Filter /FlateDecode +/Length 4726 +>> +stream +x][,~_Qφu `16xYbOʹc?e.UigÞNUF)}Tfw +R"]o?_~HĿ!`PW% s/|ڻum˟u4j^SR>x/-PIe{mrJcT0z4WyA1XN3H0䕙"^*ZϨJK`$2)+q: QGVS^s:ctQ8lt[lYbtLduEo8-]-@W`uE:8]4@"t20ii HNauQ$-N`5Rs:[GA"{AQG:A:DAt\ӊ##a%1Q0rYl18.0 M%%|KR9 fJ GP.9.0\t1=kЮ8.0]CJr\@]c0* kxHs\^X:qwuqw4xW[7swjҎE$ wA!w` iq +n "?fL +q\`*X]ʣřqVʹqb2X.3p\Q5 fjNU:J ,x~p +߄.B8.]ВL c!r\#+18$=9._-3 ++zӁwM&'@/Xs\E \>V-^B`AY' Xhbkbkx"*x.` ~w_`<;ÅY-q*eyvXn֍~35ݪRW~OF:N7Sv!CtW|!C7E3EtW3.N t=+ȮE 4r]hHYBnA8&ӥXա*&#/3ܼMhޜ'6N#K/ qs::.B4ݗ!7s_=uđ icyب N5 )jd\~7Htgn4W6]R!R$9ޙ?ApsAȠF1ΟYun[cѺI*w9;CVnYgYLݻ+#wg7CHC28s~=Mcqh,|ߟצ-{~lȝx+mDmPۊ8s:oO)Hrr} M?Q0|z:cGM<}se(sZ;qwGW|ۺurڴ/:҂F[2'<Ꮌa㫽ڼ__֣}ա>vw\$Vy>sp>)mV ȝyNᱞM>ϣ+yΆ*k;{fQ} 3鐲q+~3xav/jO\vMWW^K*+&wX`ێ+'9^Vkޙth>p ͐pto܁=C|O"#kfHR{9R7aI1f<";w9"ݾ;Yڍ|GZۧ. h_]9}$?S/׹w9R7|w̭hNvݗYpn霾 +q;><'s; a}v5\Nfa=tW}A>#=gݙt;OsG"U+\}B.o{X@OڙtH~.)'O͸1~xm=L\@ob3{ BOݫQsT+尌6PQo~p k8f6~lyΆҸrOvwN0>𘀭g s:;; خ{>O=k`w9Z70d=PTrW3S{oGmW?5醽}D(u3tb |m;9ꮟBdw9#OHw|$x^r+;=E!d%?59-R]}"h>\7ׂxN mryHf _>_qM=݈CDpcJѧw$ovUJ݃I Pc󻱙).#5Q* Jw RZw erB&t+OIvg(6`9̺헋IEWG"RARа꺐sx}/%Fe^^urs0e/*H퟈S9UvIo%FLAwp)KYr"}JcRlR~̉^{Ō9KFqzpf#SK=Z7z&],XoԶrz4q/Un̲*l&~(*J￫G46xH!i.}l^*H&}"ݒD4Y#1HT +8QXn•Pf#P]巉iPseN5p*v;Z/= Yx&6 !WRZٮVa *46h-4<~Y7. Ӝ6VGpFw^ i^2EmiJiDMxK8@g'kKFEJ5ɲy4;϶4g*k6;K=2vy* S1QFkLNpBB9g=V2XI[C1+m7(ⶢ)m|,-J5 +4'g #BЧ+e Kņg|O +endstream +endobj +599 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 598 0 R +/Resources 4 0 R +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +600 0 obj +<< +/Filter /FlateDecode +/Length 4918 +>> +stream +x][,~_1˺_ lHyB9gm:`!?_[VW_fw}ćLWIRӧԭQW_)/Z%bL)/X\dUISGw&r>寿kG]Hk宿|&5"cT * rNgT^ T1%N3\ڮUfVȨ"06(V% +fRv(-%!Zri%|L: +#|)NENgaKRq:[Vr:bM duE`8]-I.+:"T.R%d q N)MУH>0: [tgtXt gBLoԻ49آu99 +@KMr$6ȱI G:9a} +ӊ##ac`Eeu$qѰ 8.0hk8.0`[ p\`@V&qZ< 3%kP.8.0\tkЮs\`*qQp\`Fqi Ъ8+B(9.]wu@4Xv7rw V{ l48>- X 0C K8.p]01}l%TLUPg-,@e8.p]g/0j xWԜt8.q  0 %]VL1r\%qFa!# ) @B"w1' ;R뇟._~/@K~w<0zz1.ez't~Y>6z7>uiOyϴ2J"/*erc4zPCagCʾ-mXC.Ӭ4$}X4ϵC,oA]V6;5t@\5ti\{jl8;6FHWdLdN2\; SU2~gS5C*ZTzOy4*K~~Iڷt&:{cȼ`y ;W?i .K>;5 Z^vhDlcn +MDٲ0yCW/="<_ukV>QչoNO> a}=5'C{s=F8٧}<\GgwvV?Mm<%&^6l+,6}>3ocO#cyn>Iu6TzqMŐl.xk+84KPm]݊U{I_CmW{~2rm[vվ:DaL o&vV}MQfݳߥ 7l\G;>mnNyբ;tڋ It] rWgsj;Ӆ S#l6. gP!;jQn>=Gɡ͸9 )Aˡt~:o6A2kHk9la;9g>jY_nIFM\GCo8+dž8l7k-_jmڊF昙ikZnmF8<jSo^t[h#}}Ckk;{'<B]OK8yV oGܿ c@(BU{l#};>xaMŜ69ҙ =PTɣ ^g,]z=_➂;NZ, %a]h4dC>י]DJkj{VgW6ŗ! +e,=syo)ϠuV&6Ut|Pް{Z?%% <@KM<s^f)ſ:b={KO@iH:zdGCiGM9L7FhgѴ(0" Ke뮩25<<#zU0AFG]pܒ7R)Йt;M~zn+3|(̪.o' +ޟAN/~n.o/Ӽ>Mil/rz{޿ٳx7Pê~|JDӁWo"h*ͤcu#gk $fLYf/?颒|A>e\fq墉_.teֶ$z/ Ev)Ɣƕs#v6ߕlt%M3Bj-Kf1|5^vCۧm7lNB{Z]盰޻Nu_5vAfR:ܪxW*3QJ]IYvbD #J[[4fwFNXdkT1ۅM7Ֆ +5UlYS|3AxFv1 #]|]T2v+lMrqIWHW Mw*Ii6}C,2:?QRbE +):g]BoVIWֱ -m.Bl/%H[H;uyDRĺBW+yok ֶitiJ +:X:Mgڐ|n+Jچth2⢿).Sla&/}S*4Mg0ߺr-$Mتд1>K i*}饵b$KM:|CC^{9k..zvز \_-Eet,zPls\z)ԘI=VNt}UZojdjz8V;RX222RR"[meU6C|LeV)_;SaՁKU1 +Fs2z!ni> +endobj +602 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 555.123057 101.433071 536.373057 ] +/BS << +/W 0 +>> +/Dest (cb86-1) +>> +endobj +603 0 obj +<< +/Filter /FlateDecode +/Length 4914 +>> +stream +x]]$}_Qφ,k.xa}mcaPHedUVuܺ3twe>B2K +R1O'5EX]Y+'sH# lb& K_Σf9TN1ŨP)mSA&,`θ* T0UEm 3LNR }TZmu0D9g~ +pztFO!eE]NG#鬝BtkIOtx휒t%]-6X%`IJ[ ]-@WuE%]9QS)(?dtZOش# +3%(ElVIgaI6$VC*%-D'` :_b`AxIq d'\X<D&Ʌ%JCgDVV+`Nk` H\`-l1J\`hnZ uJ ,f* ,צ$.\@)K\`)qbG ,X`$.]RwJ޵9El%.;NV'+qZad$.@NaXLJJ5I9 xDL.NV5MfnH\ƙ q2 wA!^` V+q +   B7ӯ_ӧӟM)~󝦿i.u\yMg(E+bwf'ڐetǸEo?CvY}a\B1;rgy~1#?w>\NO^WaibmKNt6y"-=Ee[Nt-G-\}L;vޖ?wmtrP΄,=ʎk3qT-6UoKK~',oQ0QӼe*j?٤Y>蛆!lxmmķ'J/Jڅ ]`fS`m~+뾨~ʟ몟觙mնƈg>IB;\.o!C]zyD1f2RꟼjmSigMcqYsm">~#?cn"9r(`87{RAUq(s'Vä26}^zjA$qо[UhmE/"tXk۝=rzmA9Lrڪ5pڅ(ޝcF/ZYa23Q׵ҍO/Q qEծ/^f2SYnZ"HA<e0{VQF^-mtOPI|Ҥ K%mT|Nnű觖~C=tFuF~tXaU3/QEGۉ^ 1栘Gr-#:NyL Q-ZfI0~m 5TMdѶsł9ފjҋ^“&\f +8M t +} ++вr8. OUFO*|5HWBcNXI.]:ƆY/Hn[I˘"qEI3,#.xdnL$w+,.XA\fyUVZ 44~xxC$mޤo\FFxԖ˰ls- :xZY=ۂ>4zhVLz%畺*[UZ'F*k%jSXU'K]Wi{9^ TnZd[9co)[,Y๛ʶuuu=fnH葩&Cؙ R@h-/U)pEylwD^ 19kUz`ل7(m+:XΖؤ luS~DEj:XKAoE[Y[?A} +endstream +endobj +604 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 603 0 R +/Resources 4 0 R +/Annots [ 605 0 R ] +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +605 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 123.873057 101.433071 105.123057 ] +/BS << +/W 0 +>> +/Dest (cb89-1) +>> +endobj +606 0 obj +<< +/Filter /FlateDecode +/Length 4704 +>> +stream +x]ێ}VxAH8ȃ nF0`!SxUݳw0;*^JaU+C;st"o);z{肽Dre\#j)i|O*QEԌ{err-ȩ`F T0(Uxת3S gTR`u0Dvpht\*:i=8o.98-6 +,l`s:[ӂt(Ndt]tQ6cX]N:%`ւI9X`S:(tN'lduj8-?GADe8PA& +q0kTqsV\[ Xl1p\z: -Z) -p8.G0D ]]O01xCr:A׬.@'=ޛSk!`s\]f@' >  0 A3t.tRr\l7qAPL'8. $AqA0n W 4&8.@br\X:آ EqAE qIMqAQd @ qAMe8.f j&x̸zs\ lpQXlq NzlQp\fnِ֛@g]L q\ 7rk{ l@z-V6d <3 _Z#Aepl*Ra?fV0E<&#w%H6~|?7c,H8cj +@Et-qa9ZIKX9FFa0e7dNBڛ镍֩~=JT= P:0wfyEVpp 計B1. +Hͻ Fl͗Rd>+vOo_OIS `+JȲ 0 +xqyA!gwPʽokVg &,Wj֧ߏX0ٮfT8edY׌vȏ*kQ^7Tʰˈ(s,KB }z+K7soe1 nK]E}jQw̌6ɔTV[R}yΆT%vC $Xs.ܖYpW۶Al?-HH#KNIk|Vs1d|߷bmS|[Ȥh3(pm ps:d]ͭ5P֭'@,h˺ +7PX|?cbQؗt(g}X^r'k.T~O+e_\קö LmwT?HA͝yNIl]d9?ìViߟ3C%䳏 V>G<ۗl~eiiWD}ӳժejyi0̌O=jV_Ymٛx;v#3o|v{ {F"Zm_w9r}_[D8-ѝz86_w9ޕ?zApsFȨCi_mdֶ7ʞA3CHCjB=M\K걄h3oOhiPXvihK>j Ows:4WkAsoj{|R_n odw9rc܇smq^_wm;i87>ks:Qb*Iy=cDb;\ww崑*o o +qyNv{ޓ9jw|]L{L$_c0oOZOlIHpU@Rt^qmw1|"SwwOϧێL"xK7vSzz758SZ7e͟rBb>)?=~Q1;=tnCxhpaN:3tHIMrlXy9sN5).:e:8-˒"ͷ\Nq%2[̭ q=ѱSL'a6PX&4y\M. +23ZhKVbZoM+fGپ)t͡MgR%pQ.C2%_"qEI˔.UF\ԅⲪ0f Jrӓ +MuUALf뢓l* \`]_YW0F?$ǀPTq> +endobj +608 0 obj +<< +/Filter /FlateDecode +/Length 4908 +>> +stream +x][$~_1wt@0ؐBÞYp󐿟-Z]sv 㙮ҥTV¿4R1O'5E_I(?/V.N^G8M 7\ou}6Sh{R9;*CTO1U>:㶪3\N^R gLT fXݪ20Q;I3tQPi;QJ5N)A=Mu:v +i%]B>I:[|VZysJgSt`-&E+"l1Nt ]ItN>'D]N:䓵Ji=y`ӊ:(tR|u[e$-$xۘ0Z nt-|L-*0&%LI$`\rIbb 'rx* X[ X [F-6; -H q8+q:E u*k ,)$. \$.`\r,qZp%.`]냑vKA ޵N+ xZu8Yk8.:A`e2)) x$$.p]18a Z xx4A+#qgQb4x9- X 0Cg(l2K$.]1 cl5锃OIGx\Cx.&*+q> 3wL-d%q"nF qI+@WI$.]ВD @'8T%.]孕 wC#`MF5a +c BҁwFK:R1K\4mJ\" LN!)$.YAOԅ ^₨ ,Q[\t Ih`5Fh` B LY#p RL6UI̔"Z%t F7a*`-5% z @K 贓 f؂"q)DN ҅Nk -%.H&a%SL$& +%.0JL)` hB-X`e-; R2L[2ތd[I\U(L4Rd4NKZo& UXob'l7EMY$7N.9Zo& f`^Hތvxg0"2 MD^x}D|&+8c@seg@F7 `&V)VJ6P_cn ߢ +CĜ"52cX8Dmѓ+qGLSVi%C|a`2#h Xb,+1Us7Km62 w~Y +SXJ(RT );,"04Kv[s䈘 -|eNl;m İhFeHFJb#ԋ0ȢG3=hz4S' 3I|gGDfC3(hkVp.V풙Ncr4ӯog oQ}W>͟/˵_f8 '<ӽvݭPxIce(o+oeZeT?*u_kr-6|?{Y乗QiZ|4뿟?|w58˪j'eYw|,qw0;7o<賗 qUKw*&eq݃y *„zu{㚴6IfE`mJ"\FF*C8nuݏcync!N {aP9a@REkA& v2-F~@UhcyA_u{Y# +V^կ}$'^ë< =;к{]yTw/:g!Š g@z]ߠcA8kYT 8Vl<Jv6`K8lב87!z~wRd9 uǏ1 + 4Bf+-ҵ` K3' +\|0!y?ƵzS³#\X: s=?Ĩ q{Xۼ+1ރy7xȰʨ7#~.]bʱ~1#4ǙQl43[Ox`u8vtlOϷ{,ޮߕGfk$Zrtԑ};M^cmc`_Y)X g BhެukJOw|?}Jڡ5]¾yy 9'z74-=uc&ϓ8.uz%yV_=7?V6U;j^>dKoSigwO9.ww赽#~#Sdc1}S=q(9]GufwSCF%HCi\eS W+aRV3/<(_ov<hϭ>L٣Ϥeo؛2QK烽/uf-,yz Qnoky FYr!`/_ʜ= bpk"*`ׂmAezZe+NL`ޚcGyy_>}#>_\׸7m[Ni>-ܘE_>-}ґcEo:bx=IPae۶GW׎49)JyRl| mH~tHͻ|ǖ}k1em*.re5^͒'_ybjאTCz s0 zhXgCfyxg"==HDxޏOw@7G^|=s }t(</޳ޛ|ԑDd]9LX~G}/sM*ۺIa݈! 9w#{r<3f&][=ja;{c~X}0ӡq?vst;ݰӻ+w= GwGSEwG3=6>x~~|DxlPEږ~^Gz3 h kJI?apXa>RM̔sYGCٽgv4t/Hl^=W4WPؿjNjMy,@GtV~=NY=҆Mt +'Z4iC?$qnb>{INnű|%gZ9^Z_,*Ig73 +]v_Opع 1 ňdmз|WQ5lVKj"[8W,3識Z&(dL#M t +} ++вr8. +Cݫ*|/5HUf塱<]it _:)!=nK<21y<"Fknkbƕ\}Jmz* ݷr84|z\˰\*͍b~tQm1a7 ŝFߊNM0>ȤrT9cQ< wɝX_&/be*lÆ c:0v=^SƠiw2i52eb d>o=_8z^C]o.S>%Y}JP\ +sle&W\O;b,KuO*+`I[~ vWWF{x}v/&:HڨI߸5d-a}[M{eس-Cnu1ʤg[x^UuhtVb6%[uKiq[Lѥ?pO{+͉Jؤrע% {TOb$ݤUۭ뮳5uCBLU7);saAJU1Q;U +GqKlk<{%WkBwLZ56lU7SΖؤ luShe/lKzo!Z +zSTZ~DFYmCՕ& +endstream +endobj +609 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 608 0 R +/Resources 4 0 R +/Annots [ 610 0 R ] +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +610 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 438.873057 101.433071 420.123057 ] +/BS << +/W 0 +>> +/Dest (cb92-1) +>> +endobj +611 0 obj +<< +/Filter /FlateDecode +/Length 4773 +>> +stream +x]ێ}gyH8 Nc#)I$KRK3b<*^aDIs%r |I +7X\Y+sH lb& .ỵf9ϿH*gbT^Pe!Fム +z@sT0Ggܵ* T0UEmUf:((%`rAA=Mu:vi%]B>I:[|VZysJgSt`-&E+"l1,l-*(-|NΉ J1]O*IM+00H! +:[Yot8t ocJl54ZybLt`%0$ + $01.'Ivp%LʃJ4`\Q4tFKD`l.HL`5lq$. l ,֩%.[tpaqm +N kX.{uF ڵ. ,x:$.p]kSX`u8*J\FF $.p]wM 9h%.p]ik (Gxew`%.z-wu'iՠSxN>Q¼E/:,qbx.3I\ c2t۝yލhuU-9 +VfVʼ[yFi6S;%4>p`g4n+geHg;'?/>; k-3ᐺ9Oخ_؞,s=oˆ`w9Z7^/d#Pjjp%^GzҹũL7]#Bů{^Bnj1[(3Kgq6Y;'?<s8_y=QTz㑇F)SI֛{8 +7+k[i!5jYVD=HK; 鶏k=Clwm KYZv꾜culI?+qc6u 1@wgaz_dAy֯bQ7ƺ#~*]bʫgoڽgVR4w^ըg$L/]+ +o_v6V5y ]~ V=W}ɘ[i19WrE4b#Q/1u! jƑ۟aqJОkawLnVSOi=1Sit}%Oj%"45h'45gޮXFBZUY9"5科jl2Q˜z~WW[ΙWi)Vk07^)kjM}-aakCȖ4jQiE ^tNӹ.glC0`ѣ & +n&`9"_Oim(>fnC'eZ +&g6vn #3?eرs ݯ5:EOLcs_ {Rj'S.Dsع67$T,+7kxroF ֗?/d;UMSł9Sՠg1}- v +O +l7*&BŪkb=Q.4a5wf_ҚȆrBRT)QUfQJ}IYqD \٨4&a*@r5].$KubMZ\zҠnSxAvalL9ls)4eUb1Wɻ:䣐7P@ -+jxiRz9fU*%ЮB(ҮZ%47xI\*8--.c,/H=.cyDR i*}0V“v ˺3]zeܳiSnq1ʤex^+Eet*Zb,69[閱IX}&R;Q WiܲUoVs8:RV_2 sWk)m]u̚!yeW)GΔ:Hʿ1S j5vƸJ!C}&2B#w+X 18kUxل(U$/gJl} ͩH1D>e"}ܒӛ裦Qi9:~#o>g> +endstream +endobj +612 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 611 0 R +/Resources 4 0 R +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +613 0 obj +<< +/Filter /FlateDecode +/Length 4944 +>> +stream +x]ۊ$}gJyH`d`X#쮍$?}"+ouu;vUD^2OȪΒW% FN!D1y+/X$*{O>Z>ǯ.ZW5蔴_>eT*!5v^YǨp1r*aQf0`90JF`6^* T0CFC*!X h褝q:%'C'tZOi%sNg`Br: [4Fp:bt`uE9- +]-@W`uE8!9]ԓVN ;٠tRNԬ# + 3wLREqoqI +@I8.]В$ @'?9.p]aw`5a2c$ӁwDs:>r\4mr\< TN.)z8.pQ@չ / ,V[7`V` \ LQf-9.M!qZ95fu:9.L%Aq~"w4[0T8.T4ϜI'xIqAP ADNp\$Aqf 7q "M0%- ?l[  D```MqAQd @ qA2`\R8.Z Dnjۨśh`` Ȍ[pMsEEqAtq ě%@g 68x'7rZW8b @s+d O;#(2Gdoy|t{d2քa7W3}'k6^\jF SEW$,V0]f4"vĈ<Nᴶjz_=Z=֪䳫Eu˗* c)F*c,#͌u2σ 2eCYJsVG9OC;Ud|߱KK2kyN?[we5Ĉ(;AP6bed]<̊6ɔ3̟)FrKYL]MQT+oG<TˇBxĚ3޶2V]}Ɏ2<ؑ/pr1d~kS|)c`;B≮ˡ][GJ8B|Ȓ-} bYWC3B|?cՄbvQx2ϣО(lw+u%RA},ƞJ}?#e_קö dVÉi{/ 陠}p;!Ǟb]dǜ_ac}Ψ̇jGz;Q)i}p=+jг]Բw=ZTlFgiJxS$eosm {.!v|1-nnfk=?ѐ}7/ѷfu{2áy]7;W쌌~uՀЗ"{qmѺŭOJ2d<ح;=Rϼ&3nV|'Ww돂]![x:XYK^5/m dC恿ƳꦵǷ Q뻵Q\z'<jzk}Û7WPꯣs^;ݟp=Xw=yߨz 9")et7q/ +X>;<#k=ǧ6N|+ǧGzhۇ.wK}9MNp;P{ +-L~Q>WgL[ߩZe=S+Qut;\C}!]ugwu|"kxxk|D-fI5/}6[}h맯)Ϡu~M/m0 }^q>J|K<<\Qrh.FD'<G ~?>{H@iMu4<lߥ ayQ\y~je*<׻'}0XMMQB+1(ҽ;10:S"On :LUeѧ}렀-(O"_|_'<⁇uow}uS*ogccsĪ +[O +lRNhBU{hrk6d%]w /VVJo(EUtgTR[R1~-JY]^0 .t¦˥S +-k,-LiPס6<#Km^ՍRL{־ÍZ /ԑ/ԶqzPwVq /MJ|Yd2?SRbt MZ%46xHU1\ H[ڮ}{!pbԵҮ:<"JKb>+{(F&/ q֡jEĮrK/%mm_v/4QL&VvN )4qї$.A͸ϗ\40BNKzSKe|y1?y\rBd^pRL*:l:27 --hR_]4b/)eePYж6ĪBWIݵ>XMS].vܤMU8/t]8mJ֨.#NBqUe 3 }%~w΍StM}EKX'}Fƅ=wW{6ٕ@JUK녑/-O뜣feen䴡\ 81IC˖ؾRSdJDQ2QdD_lr9[itŁm%_}&GҶڈJKzK:@oW'ZKFUjZJ]dͲ2^0> +endobj +615 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 683.673057 101.433071 664.923057 ] +/BS << +/W 0 +>> +/Dest (cb95-1) +>> +endobj +616 0 obj +<< +/Filter /FlateDecode +/Length 4791 +>> +stream +x]M$qׯM0ۀ !5^+$#Y5=ӕ$3|| FVU;SJ9wx颦~ju_$Sp}7^W3׿@*gbT^Pe)Fム +zBsT0GgVaK*I* 뢶[U&j'`> +*`6J:Fi?8=H:ɢ.BtN!:S'I`JK:[vNI:l.uŤh%]-ɂ.+:آҒ.䜨ЩVg|VI:'lZQA +QϢxGҁ') jIa1щ:؂آb(3h"^88lD&%$0)O.F+р~r!G-UŻ 1հa:b؂X [XZ$.[oA …J Ƶ)8 ,(PXpwJ\`AQ ֵ> ,h׺$.]봒wEMQ[` J\V(qb 0p$.V&wMRN5K栕wGk28q&H\L%F]P=tJ\&D˟0VN9H\:aD]tDžy /O:,qbx.3I\ ;/㟿ӿ^/'NO_痿u},\܆z|u~fk7]0hߣ}^cC k^}V?\de4 %߱u| .s>cf߃Nʴڨz5Ů@_zTH_.(f;:`9wd2>fA.rNÌy.6*sکp<z1ޝ 5G}C'M>?<$qM}="evn<~=\bU2f:F:O8b_q]nB15^:)ss E ).{u \Z=VBM@C\enkֈ:HRx(/lF5x݊`q uih۽0D~DH;HqdCX;V/ZƉ:G}[v*oNصale7ӷ! xdCxN⼯dvx7jA1:OkwSe4j)hq3"ua5c~[vM;<]c\G:~&YOy6;q?"Ͻ'SuN vcғ;u_sU"#rv6q^ˌZf8o,>=dCqLaWaSs #GtYqXEi󽈿g~.3}a3[w﵃Upӡ +W3" +-{k:-ў{}ǘI{[kzȔCTvC? rT?.DjӸO8ϻd0˒CȱMQ8z;A GX + ^ tL 7-3{ąMt9y-p㾬ϿGjy8pbm&2qم#<7eqgos^۠|$p#,cf\zkνUd2j͛χts9-qumPK{rَ-!n{ ofe_~Uެs9Гue `w+eu;w'ݡz7Qe?Р̶1^rfPϻE.cz88lI#btFfvK.^6/, eSY{hPYnZYmuOJͥeSʒ`ya҇Ͱt \N]fV!MЗ7\YRW: 4)t&];5W^J-߂`l: #XTNjta>^1h?.N;ӳw$\^֖-[e7lXkZ^0-Kko$Ij![Ro,Q3識z&[dLd/ULBnU{4\]ײ&<.oΚMUdגm@\'AtTma0; Skr-'nҚo#[ԫnJMD6G/bҷVlwm&vG\Y +-kxҘ05;gMJS^h Eg~Mv:Nve16r餄DtJ\1WyźªWrFv]VnoSիv}kC7.m5@m2W,lnSIV~3&솁[ةβLZqo]2g7)V)(K_e:nTc Ӹu%+}N:)o6m8F]S/+:n k׫N쵤n }!]-?/@kǦ/󥭝7\.Sپ$_Y}IVQ\nL$Is=PP_0.m׹򦫬Aloi~ӼG0㾘 i&}16“\e[slI=ڲlC eԫ-*aSڭ+^l > +endobj +618 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 236.373057 101.433071 217.623057 ] +/BS << +/W 0 +>> +/Dest (cb98-1) +>> +endobj +619 0 obj +<< +/Filter /FlateDecode +/Length 4661 +>> +stream +x][$~_Qφu plcfjm~%lsEfS(RRWo D;~t[C^$/.I`jﮯ_/6EN+{IEFུ*B%aT +u)[S I\`9  +pÐWfpC{E +nh=R~(-%#hFpt.DYNyΓEᜳ|Q*NgUD2s:_3thNdt]t\6#9RաGgt8i#/H |!p:5>Ի4Y'V_h_wKhEM ` + yo8 +rs<`N+/FBeuDh_4 ؖ dT-I8.0 \)9.0`\q\`@Rs EwNs\`@w )qw%/0`GaT0x(9. .*Cq"t j!y>h8. 𮶸nj+5դH6B,,ӆLh%X> *Dq婢cuQ(g-jY("p\`q8^`DX29xW(9.] A(`|J. +CwAKB0 @RDqbx8p\:bq tLF*Sp:ThaNޕG x7 8:p\`q $%tX8x e:B8.@`-њ B }/9.8.a>ZtNA׬.@<ޓP8.\Jtz @"K q\#|AWC9SA6tS゠ A e:qFh%Q՜dg 9BE M0S:&ۖL~04ctn B/Zj 4D _q\)L4D0qeo obh\oz "M$7N\o f`M 4lk#Ӂw: p&"[nn/A#>!81^MHp/F+ $7͑ؤ286kaJ%ٴ"'ܳOmcf&#!/sZ,/?c__  hn[CC+1i>b:@c-9/?L< 7pxL0}rMteLԉ9ĕ&^sq%&äӏ_~Oѧ_%%oT Ϧ_y~/n+e,GyԿQO4 ^'Q7C +oMY6qz'UU!/i +!&H |3y_ήTDܼʩis6u,tg'3Z7{YNӡy=ţ7:Ĭ65[8ū=bޝ?ܷʑ[z61iZX'@mlN8v\ R<3auWgj9Fx%|lle4HۑY=5.- xu鐹&:{|lZF"is6Ԛ|곱x TQo }Ēϒz0_z,_6s6tȻ|nT"n4deQ_reDX?FFrqh,|ח.{߾oxTjjI +ĝvPJ-Hhzr]u[EVV/Dž4?w'KX6h==w(.^Ԅr鐷񜾛:Ӳ/Giv>-{$N+H5ė>eȻN!6OĔ{u$o9yԧ(~xz:ǎ|x^Rǁ5˽36C߾xsY;q>vG&Vx2<Nӡjc;[fQ} tHxZlB_'/}gIe|ۑ>$kX;mΆV!ڂ={D_蟙M2i Hm2[{otHȆ=g&y˚- ;mN}dyj5' +~ N!x#V}O(CӮ,'{lߛw];mNƍta5[rM5tH9צLfn>6C'ڭ'օ>ozlq֝6C%&(1hh)>Y?qPftȧ틜}}*$5z3"<{ِ6bLkyU{Gk{YZ{=}bgwڜϭ><_|$sĸ;mN{U_xj;^CY=Hj=y}Ywڜ}SܑL +:Wбx$(lH\?$"Wp7&롵l\c mӽOlwڜ wU3*zjWѦy[\q4 [絀9J7vwe7IWoObJk-tHr ur?OqOoѰVa/%'-> +D7l r!HV~zȹkHpEIѧko?T o琢Re=$=v!՚ 6O;,O_3$BSYw5~2륤iO N]?936`=̚]R{tڜHu>^^OARP`9liS)5r,ƇiӫXH""rv%FLཝ$?r.R`^ľBc9nfi ʹ_ҩ8ݽj]}0+X}ڲpSlv.XSG]Jչ*,UeosMxದҴh>U]-4N"di*ĈnF̧&4f7vd Cr1cD [n<--KPMcJ PGԧmʛ>"M8ehpgRUz)m+WH +Ms"J5,gS-<ERiOzT/fO+:Հ"_6ե/mKɔO[JH*k>fIUA]' +S.WŶ@u"~U|1r*v;_[/-rYx6 !_,]X# νcd܈{2 sXm,u7{mq\pD܎diwSv[mXZf4q\놯J칤* ZXje+[W%ym}?4p^ᠽ3uH6Q*ZthI78hO悱z6q3H?Maҵ&HlB`'&^4Na~{=.$-^Xf8t2k:ȝP9i)AN5AFX,\4. YVOn42q`kɗJִm[Tm[ʦDK@w'kKFZ"Zd4vmmPuTV3;+=w6]ց+L +uKS ԤSPhShR#ms%HΕWBs2Fz- +EV4zw͢doX0Jsr 2Œ)!!T}*8l .:yT^*654OK +endstream +endobj +620 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 619 0 R +/Resources 4 0 R +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +621 0 obj +<< +/Filter /FlateDecode +/Length 4980 +>> +stream +x][,~_1u@0ؐ`8!9gm:`!?_ukuuO0Yt.ҧO%G~/93/_ObV_JHY +'+ |ʻ }߾8kGSҞD*'QEԌ{errBcT0zZar*a +* mkUKé`3*)`TB:"ъI;98q:%'C'tZOi%sNg`Br: [4Fp:bt`uE9- +މ.+:".` ЉZlZp:)' ljVNr)ℍX-ӰӁ'!0UBKNgaRް:؂@9"(#h"^8@y? G*8P!N{рv2.z)1`:آ@+؂h [hZ8.Р[#@o …  4(Phpwr\A ) 4hW8.]mwFNV[` As\`Z +qb) Ыp8.V&wUUSƠwEP2q\LF]P=tJs\& ޕ3V q\`2X]Ǚq /OҹqbX.3p\0* ,xWjԜ+UX.q0I`~tY]D  -) 0 O"IX$qxC9.p]LVk xW=8.@\-3NB)8xWH0 +#K&Γ@ `(Xq\tXpm9. buxq)8. +h8. + d%ovy[v}`` "E dSp\z ^tNMY]Nz 7bqRS?Qq̀N |-*`*Bd$lAq"] (؂h゠VNp\kIAqA0nP 4&8.@br\X:آ EqAE qIMqAQd @ qAMe8.f j&x̸zs\ lQXlAq NzlQp\fa֛-@Mnwm3q:."Sq.Dd˭zg2Њ3;k1[܋ʳ{Kk$vfbekiJ)mE[Tp3FF 辮9Z2lWO|?E)+?BJm_a%]p_ޓ)AyӍG a8X9 0DLJ7Xc}hl&ߜB<r떯k^6'^ fY*A4&EyKZvNg2ziu%ty^y2\2i[Yڭ6vm*6|g;²~?;imuey Ƹ _?[Fn.=%Oj0!AׯmEߠԕ_[ίKSØQ݊y RVQtYcl&ّG",e DMF 3]rp6FcS.Ǵy8L]v'cГ_}>(zk0b29?3Ocx * `tMq %bg<~[k$>5p9.LMpU߰E?WmtIKm6fot85q˱]x~iv;߶Űe4t o +DC?C}~B_"EW3%ڗkMHgcۺ)Z;wc%W=scPPY8N΁=; \>c[nW`Нqq(Yv ^)G<>MT<6ΘcbR&үY}W4fRY_~?濱Kg{7PF +bA (kf9mmm6S>&Iw(3է˞ȬhL9ۦ(~!rS)6om"ۑz,!5!Yk嚍s3XeEuvy/K!b$%^wݵ>'n= ߽dK:2`~ƪIX1(_"au9kwHc/y4 7'cq{p^#)mC/G޲nv?V!C~m+uz_k*n&r\v"^~K֊Z~yg iKVɥwK~a$}ٙ#L BYI&F Tܙ^왺MIF3ӓ"*MzFlL{|eY. OqLmy+^4?gj~[(tz\1#DN'bjd7I+WM{hXj Z(Xb7V׹&]wݹ'Pޗ^*UtgTR[R1hά譳0 .trbl)_S57Sugd#BK1=^Z7j%#_O3m +a zʭ ^ԇ* "wv3U*%IV *^::4%%`c iNc1HR,+ꪘ(re/TĶBu!nv:TݛYdw[mbW^\v/K(&RI+QgVY'lkZى|lq/OiS*zjiI GY<%ĪBWIy/k ־i|K)dz}q +}spN)+Jڦth2/]Y0WR]O+4]ta]3m.Ԋ. B`_W0F=dWFI+Wk/ #i%<^ӥ[^9Gxmen䴡B 1IC˖ؾRSdJDQ2QdD_lr9[qKjq`kLѤ?}T'ھW*-[f-ξN5 Դuee=w0"SѲUʻ3vT'^ێHN'sC_MFnγW){%}Mh-TMx"^WtE[E660[Aj'(}VLBRCOShi}~ 7e +endstream +endobj +622 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 621 0 R +/Resources 4 0 R +/Annots [ 623 0 R ] +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +623 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 502.623057 101.433071 483.873057 ] +/BS << +/W 0 +>> +/Dest (cb101-1) +>> +endobj +624 0 obj +<< +/Filter /FlateDecode +/Length 4869 +>> +stream +x]]q}_̏p Ly!%%ԭ]촪Q*KjI~~/bLדSWJ'w֊tѝ C^O9>)%o;sVB A9AR +* .* JI*ADAaV` $AF)QC5N^=Lu:vӒ.M{%t8M$l!zDl11XI`w芢(Kvr)!nuFEkzruQ$-^$`H: [? +wJ-$%q"npF qַI+@WԥI%.p]В"Dԅ @'8T$.]嬕w!uF#q0m9`uRL!Jc%xW$.ݼlJ\C(q  |)%.IAO N₠ p,Q[(dS b ؂ADS҈l\Dɦ$.c?J:o@ע.B!ЄXK\1%.O9\f@Iₐ` XhK: [dK\H:%.  ]W`^t>3J\O9T;M0- 0>y$.MlsDl$.f Fb qARI)ơSJ₤#$.(qA +Mly qA":؂) &IybI\(q 7ы)ALw]3I:."S/qDd+g2g3 +밶)jĜ=4YF/f B6P_c9STpX3c'0-}sbv؜۟O}{~S2eG#@( In[TLRBHlDHH Z\nH,yae9 to[ xHl͂dؾY Va(c!T*O'ub΁Cf>Klɸ_N~ӿFNOJ:_gy\!boӅlmk9eq$Ȗ&mf}:AU~YeҎt^ԩmΟpOk@(sTJ>wǪ_zز*JW7 s.ZW+Qjpːڃw݀D;-SY{s.o*4!3 o``g)vCP:XFF#ma9~ ˯_NCۀ>et]m_tKW9 98W!hVˍ}Y1@;@Lֽǒ.1^,sY}e}a8/W܇U]R9!׉32ױ:Oգc13.2|9񸜏e5mvX:Vu=jʨxdd}˨2:φ*s={ݣa|̨OJ9;t)<RGL-B_16j{Ѹ{͎Euαem;-q0n.wCaޥRLIEӧzӡF _ n}urv}m3Ĺמ ٔ{aUm,̤H=Vz#p"ib̼}Y3w.VPZy6D1Ϊ{OJ r|ewdy:߈U?;hjbW}Ōς˭>a̱U'?X{#v=rzϬvcT"pN;h| xӡF|:F㕓aʞ"v@oY헅X^#˾K:φ\:vka]nB1xj`XփuזI +GGPt9(rQ29G6g;rL2^2uih/(<RoĪ/BXh'=s|h֟p/9`׊5bs-ua9}bgy:ľ:1;|e ȯ6w@`Cߍ8#ݓNvW= Ǥw\<ߗs.m s;8|?lǀlgQŧGtqo8ZîEegCQamf{;mZff*Ccy(Wpγ߸`q5S=2R$"[p+Ũ&)B:OԃǕCukm]SlrA9؃uc6rokjW+N կr1#T㺷u :ԔNn,Uspnj3̶bs`#Y[(:O_cFiM I<4JAmʎ=8 +7Ҍ 9hZfn̯ua=Qo2y6$1kBO95=.m \jٕu9b}!Fm~ߕ2Kv[w42t>/vgA_:3*Sne%}gE*!*VuEYj?mg}eks +uv h17Ǘ~/}{͖`|Z7ccn`(S=wKgpΞy wwyuů_aspyMyێvr^9bse双 3s`[~@Wd|.%˸-mY}n>/6eny"7KY>Cp^Sڲ}W}lG?`q!Zw[k6و돹 >+l?ǂ:4.W}6xdM/=rgs3rc;0&)5~kM*.Ka%߽8<[qʱ0ލ)Wpl5}9|k6BG6mʼo bX:?+j,r#>>й M#x6SO8KcvM B_ue^tr ė-KܴQ.3.گ0e(4-s ۯ'_I)ΐQYutq>,/Z;(cReAH1_yG*hA9w & q.UktC삽bXҺ _ZW?W&wZ~v3ZȖ Bz+g&{d#Lq aox`xSbB*]\:߅t:͆5 5n("J]jSջF5eF&udj$.'AM٨gXD\$[ZsmB+{T=MP9^]؆.DZ9l縿 7f#_N.] +*вv8.A?ajVY~x +פyM R-wf徱IzY.B,J(In[I˘dbӠkeqW-y{>Ѝ/[Vဟcپ$Y8E-VQSlef$魯>O|(l0"ڮ򦫬+0iiZ#z/&:Q{~vbYOrׇmͱ|P'oPk˲g[ӧF-hu1ʤW[ +8)U٪ӺP4LUYk1TRqZLѥv|T)ۭ+^l sgzKFAy&^VٶoZ]W3; ^mJ>̝ΐuJ՟ ѱd 5`- V{pW,5rw빒mFJxOUA7@&|"VۡjItTP9WK%!wL~Ao/* Jru>)?? +& +endstream +endobj +625 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 624 0 R +/Resources 4 0 R +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +626 0 obj +<< +/Filter /FlateDecode +/Length 4751 +>> +stream +x][ʑ~_Qφ#%1x0ƘekN,} ~/RʋХx.E%! +?iI1zE )%fYGwՊt]b CH}a__]>fHUdCPNP%ʹ!㼠z)$` dh +0Rt +f1QPEa)hV%a&I3trAPi;QJEdnt/|L&FY;@iIｋ`KJK:[&RAybU `t芢(Kvp)!uFEkzpuXQ$-^$`H: [?;wN-$%qop qA+@WԥA%.p]В"Dԅ@?T$.]嬕wuF#ql5 guPL!Jc%xW$.]6%.&J\xFiH!x |Ru~ ((K +$q!%.Xc$.e4o1<-l(qlH<1ft-"t:H\ ؊E/q~`wI& B-X*`+g.4l-q<]贖 oPh"L$.XC-Hq%.v%.2A u4!q懃m'qsDl$.p%.e$.69 $[I\M4R$o(qA +&Bs]Z޹CX'g>dx!TQ*Tuv6Oj]rÃ5uz;;{÷!]%}r5T~˾obAZtGTx/sT0Jm(A9Oۍ(-#H]=S]ֲóp y a|6ϛrK'xhf`ӥiv}Td'͙g2= =׏Ee֍OTekέ&b,:+{4}"0gH)y'g7Ep<}0|EUBW&X(幟jeY%}3X=5rΞӪb[X&m®{_FA>:^`=WI٬f y76sTQ?΍8*#e⦘Y!r1E{dNydRڝ3fFoE̗$7fҗ/ԕ}ڞK^3,ڭ, {C:uelC{(Ä.v2/lQfd;يY;j +;ޮ3mg2+ڴ΂msD}=WSݍjw@u#\C*{Zs.qb+ĸmn_ ,=:q%MywvLu{c{ܳ*fk1:O}-a5]zd4;}γ+82Pa^-n.^ֶ5vrO-T:Ssu~žsyfO׻vfw7R+>'WWfG%fV/^k2WOy:dk=~olj(#xsuw[p%mu.]gb=}Hչ\?Y'U`C.CfD_M\KhR;qŵwm̮^4.:K۾tZI ;su~ƭ2u]kB?o.ʱk~}b++ziRw?>57suyj 3;:]wtW Moe˗'N;Hf3jm)c<)Ϡ랼Vs*:(ȿzZ?ee-eUrthrg#u }᜿yt(e[c{VJGhQNy:g*"X";#꽘3?\g?v^unvOpq1k5G87o; g9boNXqae6գV~ˢ/yڮl)u∼|y6׹:O99#.U>lϜŽqPme&ns+3V|ȓu Q*q|lf9-@? ·8Y{~UUZh0qNrgV9@\yWx`[hve㕝>.0[V<+jLQGuяmZXdo~S; ?s83=p־a\9մ|:B8hoz8ҕ%]I׍a?R?D,OLE'xyQYM9h 1`NY8}T-ҥ`aH˒pf1?WΆRIacǂ5^ՠ'16+v:~j/lڪNBuĆ=_P7a5:M)kB7{ڥMzJSet2ԕuBΓ ]w4M̪f_y0 +crDXnjՙ +TlUSfJuYI=Q|=@k;ܘvqTWb]\JB۵[Q54)_?ai?0U)3OB_ЎZ'6x>HW!i4V.Bl?^:*(R3!:wa&5*z!~_إF+YgDJ 1*h~lX5U^-WKJ<@H5[&5(sBΖ J~M6K^ +endstream +endobj +627 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 626 0 R +/Resources 4 0 R +/Annots [ 628 0 R ] +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +628 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 770.673057 101.433071 751.923057 ] +/BS << +/W 0 +>> +/Dest (cb104-1) +>> +endobj +629 0 obj +<< +/Filter /FlateDecode +/Length 4941 +>> +stream +x]]$}_QφG 5x0YkmL_>#eJTFfe5+# +*}UNHz1O/j +N_I(7V&C0L׷ .y~q:)%okVB A9AR +* MLIR V`$ c0RvJ0M +f䂠 +vh!At/錞|L&FY;@iI&ｋ`KJK:[&RAybU ;IEtEQ[WZ%;D]N:䢵Ji=9`ӊ:(t\uYe$-Ip6(VGUVK:[ $` :_ba +^)&!LE & +J4`ȧ U:%" +8X [#WHk` H\`-l1d%.`[Z$.[RIK\`paqm$qJI εN ,H=H\`y#qZ^ ޵޵9El%.;NVG+qwVAb Ыp%.V&&*&` +uax84A)#qw /q2 wA!N` V+q +  xW'L˜[M:&/qui-b:Ѹ.^/74:?_Z:754X(0^]uҐ&#t! =_yNڧgYc9u&;﬷Gy/lD5xފ`qA¤4M/ 6"һ#ytȱe +=>esڗ9u[Nk ]+N1p`K煝6W 9Z<bړywٽYsy4.kF8. z7jAs'NgwйY)X$xA2IONy:$߉c{r6qˌZbaE}ŷ^P1={nzkX4qXԍφ^:Ϭ8^߳|@;@_HeGݺvP(=Pz"TO0{*Z 2i+?6,FU4MROy6_wvV^խ}$'ܻ۽`Oy:Ǯ^()kpvksZg]3C40ּ2D!;B5{]ߠcAM[#+eFy`k'g~dOy:ǯ#;J$ϐT=?Cw@1d|QAf\lAfW2pe~m< zAr|y:$ǸVXn =e5pew^7աUa=h,umӈvyOy6Lӝ]a>szk`XqFe;}7]bʱ~޻;- Fc<#a +V3 +oίi~y:,vmOtԙ&B m +K9eUk|{=XOK6%/kKOOOwhkY <<ֆ9R[6_oqp٣͒~)fowFؖUXgȏ, -3v;vu҆?*ž=,=e[o3"_L+\~O.3z}X|e[U:kSՑ1١z[V}d-ؖEOBi_0|]|Qmr,s .R⿎%`Ó^ql_k_>jCqV+q|g5oK|f"|4UWƂ^C_/Ιwr}݃;xԚ6]Zzᰎq rRxq^0`*({6|2Pi{ .qUG$ }Q#S?~'&;>zzn6UFc1pņL,jXL\05[ҔA0gL9k=w20cFs(Z(7עαF>i˲,eXڗ{ǔb +1'=IPeyLh;?4>HzX#~Pػz2!>cJr;7okN@RGҾ]Y򤝑{;Z+G7vIɅD0C:]Z_,,E 򪈣]v+@#jenOT 4nRAސ?W[-H&ùbA^oŹjҋ87Cr)Cw*&t[bujg6.lsi5 ; MkQջF5eF&u5ej$nAE:Ysuga@.tv-S=kt/n쑜leǵ8?5ZE.Hⷋ Bj׹B +=.&KCЬ|*i +פyM Rd~+cNrX$h +=-saKG%ϤR7eLR5/T#X75qƕ\]Jtm:*ݵr7|z\c\*Fq~tQm1a7 ŝFߊN,cdҌ{R9cOx(K_i~߿^z<4n]L#'̩6l.ֱu m75{MB}ĦAh<-־|ikj.xsp)ʊUNڧthɰUe+33 y%~]as]삅vi]e, B`/xyie`}1ڛK[ò,ү/ۜc˃uv,{9}hЂVL{%R*E TC/٪[Jb. ҜxM\nZd[8Co)[,繛zJ[eۺi]wm =2U=ۤ|;saAJUN`zLԤ5|נރC(]LȶFޝxd#RJxM莉UAoMx"W7Jْ>[Af'|mX,B2b@ޘ((-woaI?hl +endstream +endobj +630 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 629 0 R +/Resources 4 0 R +/Annots [ 631 0 R ] +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +631 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 311.373057 101.433071 292.623057 ] +/BS << +/W 0 +>> +/Dest (cb107-1) +>> +endobj +632 0 obj +<< +/Filter /FlateDecode +/Length 4702 +>> +stream +x][,~_Qφu am⇁Oʹҳ0އ)+uIe䥲aOӧ+#t>} +E*K*!zG/_/Rx+oH1?۫U{% >Z6/4j^sR>x/-PIe{mrJcT0z4WyA1XN3H0䕙"^*ZϨJK`$2)+q: QGVS^s:ctQ8lt[lYbtLduEo8-]-@W`uE:8]4@"t20ii HNauQ$-N`5Rs:[?GA"{AQG:A:DAt\ӊ##a%1Q0rYl18.0 M%%|KR9 fJ GP.9.0\t1=kЮ8.0]CJr\@]c0* kxHs\^X:qwuqw4xW[7swjҎE$ wA!w` iq +n "?fL +q\`*X]ʣřqVʹqb2X.3p\Q5 fjNU:J ,x~p +߄.B8.]ВL c!r\#+18$=9._-3 ++zӁwM&'@/Xs\E \>V-^B`AY' Xhbkbkx"*x.` K,I_._~F_^/NӿV_Fy_Y[9Mg/ְ~e ;M*QwS8 6.]uo]HdK;e+4mBK}Y+kp]PނY +Ŕ6?.{#Ïzs>foC;leC2^1&s˘yi՝yN gwAUnr|{ټ:(m7P3P}է}Çf^oSdphc\g_Fg՗ܽڙt]?n9GwAKhgdt-t.ՎGl~ڴeoOMoehNJQ/PJ-Lhzz]u{E^V/rY_}ڬP2-bYOVQG[]ַp/9~};L~Q|3T#: FxƿA(CmEv9F r[Nn1?; +Wd6-K;m[w(GI.v92qwcx+!!Jmt_ӡtÿܻhRvҪȏ]h(BXںvsÅU갚bpufIbI~RzX&m]:9Xmڗ{iPu-RpGްy`Eڙl h7}P;.[﹏ΎzӔFFL+^ٿ*Jpgӡjc⿘wwqDmt 3鐲+~3߸{.Dgյϒʊ)]7V-ض#.Г=;9ZkoG=gO^|DlI[@jٲZHK.jѰqFy|7;F>_lvs:D{ȏtw97Ӆг>l嚲jܛN+˺Rc{yN}Opsqރ_W};i8@ug!wZ"I{R\3E'9}w#IM~?Ǿ#xν3ِ7|w̭hNZ m煅{KmW =s:>|~iW;qw9}gWS߫S,Ljfyzfr{>#s:}٧G"U+\]$EXQhg!9<sH~XAo͸1~xm=L\@o}b37=jFEQ2B"~wܰ6#~jFjcPz`_> +endobj +634 0 obj +<< +/Filter /FlateDecode +/Length 4899 +>> +stream +x][ʑ~_Qφ#%ˁcX0Y>L,}࿿_$Uw{(4=]KdD))K_~~/bL嗋SK.2 䣻 ze__]w]G_͔7]YdCPN`%˹)zSX2eA):1Ș("İݲ0AĂ: (% "Fi7y(K<'I"/xN>vZ{#Aă,dcxXoȃ,&+d1Nxȃ,+-\D"/b[QnrZ%|ӊ<(g W.<2BN8bfmAc<K`\e%)3uG?2h1daV`m?A9$úh'pyD~ +#(cGѪo>].}GD4wO?}Oxৗ˟M)_7TAqʹZFrV/4Y"mr;fyl׶[Q(ȝ:_~)XRfu12_aԗ A_?.3쒤yCGC^|{)=+% R88kFʼlaٺbN\,c;ǭ=inBcq!3DVm A61[~o~Hw,횵aZݝ5~tn-(2eYM<aiw_󱥬 '۲||3J}QYE2LsMS.<:Yf^6~ JkGJannW.sԬlaՂC~V-^rيʍmRnlW⼊.mt׍UwZ|%͛/ʴ~Aݷ]glm&vOGVKU׬IoYڕwVe\+;6ۜmS:oJ?0VuUDSDRkU f#0I%e5tXK?i?hlEZt/G);ދv~J6jh7PC&?:E;oC8> +٭cnrѴl;o<ˇ|yT 4O`EBu7]ZǘD|wz^^p wN̾zu0K+,蝬$yz>~^c~I2$ШXM_>>m=Ctc=#7;5c{gxΪe!~Y|oKSJ.=twoZ>䬁)͖Msuwߤ 0_<\Pwo~+[6xcI/;WI۲l}Q{0ٛlej"#go{N=z6G~M+2+Tzj2{:^!luځ5& +k36㺺|Yefy։3i_u]Y;3Oqqߺc=!? Kgtiw1tԪzͭ>?!}gVy/l93)G=f y:/</VtUFV|[J#5Zw WwQ}YW8)mA x'<Đ%Z,Y|ڏ-lpGZ#~M ޜ;D7 +iz{:q?w\{zWzǺw6?<8}vgo}*x#1b)X:b1߼rpG4:O'īE9+J &~tG/={>_SZ=0RbNJnBIyukAdoW£74n~C,V1~oyqMz +bxoTۋ#2ss!gҤ˓uf ]W{>~;ŗ!N?<쟾#<][o{a-Sw=ϟ22N߆qx8KMٌdsѽv;9b={L-VD 4EQNy6/N%iGrH7F.iQPaUnж;S`o kq9[)8hi;ȁ #gz8gN CmX*>L|ϯ+Em%G-,ϬI~p跦k~4-\֫y|m=NPk밵j'ݴCy r{ϒ.fYzf>Zj9^|x|}w%N49diєF~$ad&.Umt=[n0uZ>(qe4\ߕj|R3"jz%դ2'\* +YA˶[C#-Qu5"<|#V{ 9 #UХ]*vBJud5U; E JcV~g5>Ys&HT˥RN .b5W-jت&Jsu>>&9l&9"*v~c6QzqA(V(W\'ڮJKr6,4>U*#O-ڮZ'<7+y+ꐤMԥC>KG%gЭVNe="T5YWd=Q!]#ʯ+rթT]m}מx2 6t|OqYGlk&Zщ|lLZqo^2g3|=,}sA:nTE SuEW2xߚQutDC7+Pqc:V]5`/%uEP]=5vdS]hiok ^۴{ep|VZE?\ƳmIvM⊋%\2]E\05VbCTճԛr_EFm?eW +endstream +endobj +635 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 634 0 R +/Resources 4 0 R +/Annots [ 636 0 R ] +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +636 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 558.873057 101.433071 540.123057 ] +/BS << +/W 0 +>> +/Dest (cb110-1) +>> +endobj +637 0 obj +<< +/Filter /FlateDecode +/Length 4913 +>> +stream +x]],}_QφI1xp1fL,=2%E*#ީ1{]P(R7%r |IM++)VEF?Hv%OvӯN_g3@*gbT^Pe)Fム +zBsT0GgVaK*I* 뢶[U&j'`> +*`6J:Fi?8=H:ɢ.BtN!:S'I`JK:[vNI:l.uŤh%]-ɂ.+:آҒ.䜨ЩVg|VI:'lZQN +QϢxGҁ') jIa1щ:؂Ηآb(#h"^88lD&%$0)O.F+р~r!G-UŻ 1հa:b؂X [XZ$.[oA …J Ƶ)8 ,(PXpwJ\`AQ ֵ> ,h׺$.]봒wENQ[` J\V(qb Ыp$.V&wMRN5SƠwGk28q&H\L%F]P=tJ\&Dӟ0VN9H\:D]tDžq /O:,qbx.3I\0 -8D̙)R#3Ct=I6㯧_`=_ n[E4TW,!#bFjFFb퇕0LGn`D# X3wirD탖mv_Iޝl[3%-R 1&h$ܙDfCx]U|:+vGAƟ?xw?}z=?o4M_wYrk:C,h^33<׆tkIGy[^Rg.#Ӿu`ŖeiL?O;2%>eDze_\X_:3(\i`x]cY9lbe4nD<(!PHZ˦ٿ*(!e y:TCrg"!p )_s5 ?R."{|a%L0!FL3뎳~뵼5C%? ˿;|#Vdm)?X|,f|ǡPsXZt{F͵6Q CD{e4y64폷M^*k ?_/j/ } r8v=yX?B:c=Oc[^/촍eW2b<φXw#|q#]-ތZv@{0ӡz9r֑Y.o=ǸFtHA<nĩ{0ӡzzuճZU zfFs:]yvuq: jBY~gm#Y[(=كy z:׸M + I7?,y__FK:^ z3[Q2و\n[G7;:l^`j|Pҳ[3s:WDZVm W|>8a ԝcX)wtx:bO7k;rϰUs;]`fS`Vw̷7~lzqkq3%WX._W9qV.e7WF]m q@1F8: tr`wSܔЖo,\ƥ^(ò m +sT# +o.P[%'p.Cۇ){? LyUxk=4rΖSs-7jk6:ةȖr!rSդ1'Mv +&:3koUU{4\]Y״&&Po^ R5,3*7)[OrߨZCg6=5:} +$N]Nb4z*TgݔuZ.l'һQ~\iJ 7f#䣐wPB -+xR:qYet5?SoRbA):_Jhl6I\8ŞK0c+)s\#j놺&& +i\Ʌ7(N߆W}+C7.5UeK(&ZIf^v@ +]ihmT[>A&͸Ne3Zq}> i~=9ϰ ?e'DLNGkly5J!C],ĶFs}d#RJxMY &Ao+XQr&a+ӜL2ʆPTZ +z /*-ȧ j\ +endstream +endobj +638 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 637 0 R +/Resources 4 0 R +/Annots [ 639 0 R ] +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +639 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 108.873057 101.433071 90.123057 ] +/BS << +/W 0 +>> +/Dest (cb113-1) +>> +endobj +640 0 obj +<< +/Filter /FlateDecode +/Length 4683 +>> +stream +x][8~_H|)$HaB<BH }JǗ8N{vLʗsY"!D `/H\# )/qO +:|ve:B$.c=$k #-.` "SL6D$qAp1ft-"t:H\ XK ^0:M[0T$.T4D\i'[xi-qA4ޠD234֒hjIGיQH~`WQ 4&$.@bI\X:bI\f Fbf )O₤nq蔒 ! 0.y%.HVa%0n $-.-p0nIXo$-&{^oF f` ^ D^o1\u $LzW ^+(F anC +܋*154XYF/F 0B6P_ck Qy!`Όo2aXJk|_;3&}z)?˜g ޷[h '-WR5J fSB$ں7;l_}:_Meci9AfѧN?oo\|z=}TʽKL.0\?8pvc+_ϟ~ixء/46-%-gA"X4(Z־33m=51nN  }äShwʕ_\j +F!n ӷ~cmi鳛^.ӵuTP.#ӎve|!Rl򰬯k=KFH*@]un5 M`5K@c'5/t%nBzgawf[XM7\Esw3ojҾL~k#uxMil7z,fuՕqxV2ʌ1jv9ʪٰu.ԩ:K[^bh zFν6C¥c׹~Ϭ=6YЦr:T7z +RMt%hJ֩%g=3zuX .sF;22Q'F^͵>)OZyOB+t9rO292MOp|5} j^WE + +G ʑgF 5HX㬗=p ={;u5ue2/l\uƓ.}l)G#73ѐg7|Ⱦ'ükzv1|RZX4?cmLWćFw^O>jW IE1ݗp+s ^ljw-Z>j|3n2̂O[<>䞕ܚ9nw܏}ya|g׼>gT׆;}λqG;ush&ugn tÃf炘#:Ġg,my_ep+5>)/Aq?m4s84cמ=z<ѻ7WjGxrd{+E^k1S +2Ww927wUw7<&6{8Mmv_áv#oxϪc\gb=}Hѹu<ߙhuqםύg͐=萹&.%XF4M32:z:T6fϏv|/#[ڲKVy ?UhigBoQW+=BK pl/TbEO}41]GR}yv1օ=g~GJдhy{tI_mDmR[y>t=jcanSAǗ+ +'̷ӗ>7P֏]]s4}ֳvV)$E_gF +JGh(;UG[mƫ^̙(Q릧;.k[5<ݝ>\Zue ܛ3 Gy`Xrt'0޲w~}iQUeѦ| \?ْ9+{܇i^|s8|mxXqWwYx}tvBΗlLW2cvGs4D>ٿ*sJpgájcwwiDmၨrAѽy _?wժM|ƣ |ֽؙ͓p]o˨]+oD{ 3И6ӕЋo}5e!7+˺R_Ǯ ods8{wN?~2i_{3AN}-} ֺ3ᐻ3]44qOK#}UpݝݕG~#"={g!uW,}{n}/s+z{~PLGìVm@<3s|7<8v<«ěI6 |z~I"(CL/O0WJkU޾h'WucW]}]ߥyoyyʉ+39??o^>>+Mu7^r.â~< +=\2|C:IQYҹذs<ǒj:#RQ|dbJ?t۟z*~;y~$S*e CSZ7c*󋠯w],ʉxbA^/|jГD|1l3@jkMP5Ul3tJ*,MiMhrBR>-[ר"QH]NY:J? I~rф,<0f QIr:At9\PRŚ5喯TX^m4Iڌ1&C+UlpcQUvrAH*W( +mSnGUR C34JHy +DzU+YFj徰I-)ԴMs +fQ tKb3id:UnۙϦ;|Vma 5_:ִ.4|M5ĞS h[m؈MѶiwCxZ82M6|0R +2VS)WNВi2-MZ63VR+,\4n_ڪ:ͶEgxMh &VfF?<YrIAgxK HXSK<6IIX0I#m ++ҤR8.`HX/<$ya!i ƓB*a½' 0D (HX#M)/aO +<|"OvdOpj84)ο[~ic er=pfovn,4"}n1ç>([ٮ_Em;GgLW?z?ZTgq ^-6k^JGG_-#3ZPqӴՔ_^y-&[pj 46 +_Ϳf6n%ur +ӭ1?FɆCC;X ;K޳7Xx蹝RG=G}0΢";k8xj=5tQOƺ#nn]BGmFr[{la.p5{hʖvgܑ8Vv1Z^wȲ?fٹg~Ḳf\:Ƿǯgfo.ߵY[/픾rK/өQEJF*ewF1>˽{'4_sBgx,#."[̽`R^]5rKmwmx0G2́˞8֋{ދ-J@E>L:Xd5g>CA*/!R <eɎ'qMd{ÐSY\7zBRyb3)y .)I7gW2 C]j 3C0uH?b`]j_/ڈ>)UMQ`{gmZ&~\M?'26#Nt.Ʊ&/o'k?΄%֟w͈~2(@7M9jȱxG7%6e;v/r#2N=uViuP6Cܬ5\Fz؀"l  TPl_Rz]j*:ߒ'+D(G'!)ؠ\| h68fWiպ­䧪֜ $N -T]= Sz!ar*b7a w=NWm8}.pӠbԌ2dmPC~]U WoxW' A9U>~Z>0%H^E,OyWkdцWqN%ӕ#\|ƨIJ['mE[}P{a)N'\jf~xW0ɛRzJngnUagDlk 6l; gVqwHιLo5Us ҐYD6O9V*ϣp>EC;[" +NNidl'eA!ZNB m8.`DR9-r"F=~㮇ߞI>(M)A0v%m +[B/SY,Ͼ3+mnQXx2-CmEj&o?3_TZ=[Dr~ECE2l|>J6|$ß8;%]IgN|/Iq|Xb)Ko,29rȯ7>2QۤvBfUնHם +SKZ7kgN Wz]3*l>h@DMd=g).W| e[)96UwF6T|^Uo*z: )kB7GǍ㳑*RT.҉BPR])Y8 M JQsfb,D\*&[ZsuB-[T4DincU@{tGe56ֱFlRڐ_o.] +D۵[Q5{i05 O3TJe_K5> ]ܬ䵒zC4VSjtTByZjlETC*1HUXuJf+XUkdWMuEnr:U73wWmd_':⚏uv;/Gvl4cI# +H1z(4rL.S~|&q߲3qЈ^)}2?otc9oFU0[Wޓ"ϸ/:m:YVk~ܘUW KI]TWnOtMT͗M3XoK[:W|b +}wp6|m.\2]E\ⲻajdoRUS?]ta_ƣf -+}ä<^<ȾVhמZ; +x2K>2yj[{9mjЂVvت-\S*Mp*b(2ZQKqjLѨ5QJtMYnx-ޜCj%k,kW*жwݠ&ojܦ=sg3:HV+!:dIkl@*z !>=PF\xd"RJ;a8&VIt*Jإ)@X +j06+A|W=P *_FcAi@~!O+m +endstream +endobj +643 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 642 0 R +/Resources 4 0 R +/Annots [ 644 0 R 645 0 R 646 0 R 647 0 R 648 0 R 649 0 R 650 0 R 651 0 R 652 0 R 653 0 R 654 0 R 655 0 R 656 0 R 657 0 R 658 0 R ] +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +644 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 416.373057 101.433071 397.623057 ] +/BS << +/W 0 +>> +/Dest (cb116-1) +>> +endobj +645 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 397.426036 99.933071 378.676036 ] +/BS << +/W 0 +>> +/Dest (cb116-2) +>> +endobj +646 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 378.479014 99.933071 359.729014 ] +/BS << +/W 0 +>> +/Dest (cb116-3) +>> +endobj +647 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 340.584971 99.933071 321.834971 ] +/BS << +/W 0 +>> +/Dest (cb116-4) +>> +endobj +648 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 321.637950 99.933071 302.887950 ] +/BS << +/W 0 +>> +/Dest (cb116-5) +>> +endobj +649 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 302.690928 99.933071 283.940928 ] +/BS << +/W 0 +>> +/Dest (cb116-6) +>> +endobj +650 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 283.743907 99.933071 264.993907 ] +/BS << +/W 0 +>> +/Dest (cb116-7) +>> +endobj +651 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 264.796885 99.933071 246.046885 ] +/BS << +/W 0 +>> +/Dest (cb116-8) +>> +endobj +652 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 226.902842 99.933071 208.152842 ] +/BS << +/W 0 +>> +/Dest (cb116-9) +>> +endobj +653 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 189.008799 99.933071 170.258799 ] +/BS << +/W 0 +>> +/Dest (cb116-10) +>> +endobj +654 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 170.061778 99.933071 151.311778 ] +/BS << +/W 0 +>> +/Dest (cb116-11) +>> +endobj +655 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 151.114756 99.933071 132.364756 ] +/BS << +/W 0 +>> +/Dest (cb116-12) +>> +endobj +656 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 132.167735 99.933071 113.417735 ] +/BS << +/W 0 +>> +/Dest (cb116-13) +>> +endobj +657 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 113.220714 99.933071 94.470714 ] +/BS << +/W 0 +>> +/Dest (cb116-14) +>> +endobj +658 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 94.273692 99.933071 75.523692 ] +/BS << +/W 0 +>> +/Dest (cb116-15) +>> +endobj +659 0 obj +<< +/Filter /FlateDecode +/Length 4421 +>> +stream +x\K$I ׯ3aKB+J4bEZ8*WeNOҎa#3˝-~qٙK|tdM +jEe)d+1OBdR > ./?mw]8듰L)٠ +X֑ +CTX,1BbϷ1䠱 {V-@ k,JH +Yἵ*Xf +31_T^%LLxCx YBNx)'! E*H%⡝2dueYlN2dfW9[Ld5s&6II1)<Y EA@k<,rVՋw%d>ʃ,| |,6E AFpIYp次pJ/p,I<4 YG AjXC1f]@Hհ 2.aASyŸ+~ -cѰ wHÂ|f kX5pb5,] AqH4,@|3tU^1aAf K4,kX `aAlJJQÂX,xHT^4$ Bd<‰5,@bm +aA򐅼װ yȂAd8djyY`.j}@2 )DcЪƋVy<4,H BӰpz F% x5,HU4,@(2#3x ְ.xiX=dA6aAUf B5[3TQ *M ʃ, ?Rl PD ְ g/fְ gװ)fҰXȂOÂb4Y8xjXP\FYԛYÂB&Jo.+e/f^UȽƻґ;~3T m֖!:Q =w}VRz +(sM~fIъ M?@{3ryG1:E>>^ ;'YGSۉmԦўyݑ8o:_ʑw9Y;&S#泻;DkC9=aL5,Gnby?m@ǡbBaܜ؀C&[bQ܇ $o{ae'.ȑ="9Ya[ơ.0<okag6M{VJדNJ?(ˉ3-MpnV,.d|Q)eoRﵾ6Kw&F"vg;KKNQR$;xb̤obW >7Wzxy5x,6R<蝇̲+-- /| ru1gId}]ɚ‹K'm#[Y!I4w`ƅ.o&c©DG?m6=[LfK/ +|ϯ-Kgy2i{`y +d)j}|%UZ>rPtWXM>dirVzǤդ6 si.9B|C9Kz| ~<tVyۃ6* Ob WO6qgX]lhIdu5֖ڿf;#7q>:R;Y\+jʒ4Qmo'l9)%r w~.<ُCzBҖ5T^eV9CjqrTK^JjݟNzy;ĢyEAo#jnVcD:Ki"iD&NdsZi5m%vJdž ,'УyHĚq;N݁U0ȑvꫭOI2gsX[z.V|߂E'wyWMYB=Qѥyr %qOdrSudumM27{Hm]GJ(}:-mup])zwC}?]۝}nk.r<=ԙY3L[Yݺ6O~+yէ6>odi~{ځ`Xe,H_|@p Wti+~ tzZok[^oӬcֈ#r7z;ƮcuNlG|6ЛPZo4~s_ƅ/}uNz `~g.}5E~*|߇㰽G;WCΩAUw6|ϻ9Y^y0FU!gZ>R߰鎆qZ/XQؒlig/ȃм Ťsoy^ݷͰcq}~>僧d$Z}]ݲٞ`H (LwMSøZze*ķ<ޭ) KݔT#gGo9rw=u#9w\vgw5&5|{m?b}Gdmy" TF\*gy/+[b׶UźG~/HV^R6kMl|[ɰpM~9ɷGBF~},vp.{ɔ-v8N $;jhj߮旉"zFwK[V|G6^.`w'yn;w [dyc{b}ViG 5E{iNIҩ}mhfvkB͐+BvhTZ()w+tJ+&>nr(n r +=q_N!) +Dl <|k6{&*S?8&%1f'o4cG48\Jfg}dQ ]cꤲJO@ǖu@XOʙhnC7 y'n2رe&N^q'oK l|MmA& 0F_*ykG'q8q/O5y +^ +8mߝr^\⸷-| ydMH YLDˣmh^n#ܙ'M +-DKfДpT> +endobj +661 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 799.173057 99.933071 780.423057 ] +/BS << +/W 0 +>> +/Dest (cb116-16) +>> +endobj +662 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 780.226036 99.933071 761.476036 ] +/BS << +/W 0 +>> +/Dest (cb116-17) +>> +endobj +663 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 761.279014 99.933071 742.529014 ] +/BS << +/W 0 +>> +/Dest (cb116-18) +>> +endobj +664 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 742.331993 99.933071 723.581993 ] +/BS << +/W 0 +>> +/Dest (cb116-19) +>> +endobj +665 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 723.384971 99.933071 704.634971 ] +/BS << +/W 0 +>> +/Dest (cb116-20) +>> +endobj +666 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 704.437950 99.933071 685.687950 ] +/BS << +/W 0 +>> +/Dest (cb116-21) +>> +endobj +667 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 685.490928 99.933071 666.740928 ] +/BS << +/W 0 +>> +/Dest (cb116-22) +>> +endobj +668 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 666.543907 99.933071 647.793907 ] +/BS << +/W 0 +>> +/Dest (cb116-23) +>> +endobj +669 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 647.596885 99.933071 628.846885 ] +/BS << +/W 0 +>> +/Dest (cb116-24) +>> +endobj +670 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 628.649864 99.933071 609.899864 ] +/BS << +/W 0 +>> +/Dest (cb116-25) +>> +endobj +671 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 567.702842 101.433071 548.952842 ] +/BS << +/W 0 +>> +/Dest (cb117-1) +>> +endobj +672 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 194.455821 101.433071 175.705821 ] +/BS << +/W 0 +>> +/Dest (cb120-1) +>> +endobj +673 0 obj +<< +/Filter /FlateDecode +/Length 4933 +>> +stream +x][$I~_–y"00g"g_d%*̸ٳ=/̺w$r |t|RS +>џrq:ϟH`71_Nguȓ~'41gI3|tmUfΘ$̰.jUeav +f죠 +vh!9kSӃ3z +),"t:Ig^K<|tk甤-Φ(l*Q[LVEb,l-*(-鲝|NΉ J1muF'kzuQ-A,`H: [$xۘ0Z nt-|L-*0&%LI$`\rIbb 'rx* X[ X [F-6; -H q8+q:E u*k ,)$. \$.`\r,qZp%.`]냑vKA ޵N+ xZu8Yk8.:A`e2)) x$$.p]18a Z xx4A+#qgQb4x9- X 0Cg(l2K$.]1 cl5锃OIGx\Cx.&*+q> 3wL-d%q"nF qI+@WI$.]ВD @'8T%.]孕 wC#`MF5a +c BҁwFK:R1K\4mJ\" LN!)$.YAOԅ ^₨ ,Q[\t Ih`5Fh` B LY#p RL6UI̔"Z%t F7a*`-5% z @K 贓 f؂"q)DN ҅Nk -%.H&a%SL$& +%.0JL)` hB-X`e-; R2L[2ތd[I\U(L4Rd4NKZo& UXob'l7EMY$7N.9Zo& f`^Hތvxg0"2 MD^x}D|&+8c@seg@F7 `&V)VJ6P_cn ߢ +CĜ"52cX8D}1+|ge6< R4zd!/'D+M װ]"QAb_rR4b͚Ƥ&~9aJ}O;0ђ&U H~DGe oT"5ʊpӪ`9+c]1ԭK eWyNaneNX(C("RL6x !.Gҫ4m +R2PB؇^rB]:=ޑ{ydRN?m$ȫm xc1y#~|om9Av_ye@ 6 9( t&gsʢѝML0\Wɳu]>T[= +k{A<_VhmiuQh7;[#: g5I4hhvNrK]Vi[O^)JӆU| {4(-W@IK}/w=i`J;lhmxD-*߮Z~jߡ~7kd{(kZTQ>EW[G tQW+朞3U_e-?y6ᵼZtZkXBiYx% 4nvLer/p<'|jUu遷c +vR2wI ԜW٥} dOڇn#W>v3+1"gz Yk+KuO8}e&i務|8WJc{$(ЅYhG'DQ+賆F4^Td贶BrxIV~JtsYD.uҏe W+aR.pZUئ:Uk%,|AJϷrT AVn3J]؛2B{烽!qU3/î[Vin&yX9tB K51b}~n~_k>`Z$kk"9Dۤ1y0@rיs\UZX +a岕lt%I `/Y[Wx3=T(>.ѵ"v5nFYVΓtkVjgٔFqc8Arys6h2M9{:XVqx(K\y շҿH |.QEYχo7|f ڒȱԻ.kh;A6~7U˞j+ֹƶ%˵4cQA ۚ/Vcmjf I7J=%.΋z:zPw%KYsX#4m7vBy4ˇG{c(ynjhyV3|'m r$Z{|0!r-kc(IiE,B5"宣Em)зFYwoI7|$x0!y?k5O%u4pYIGMV=}efN#f$.ܠIX}ÑWDWК^<Zcԟ.{6[^d$G?4@3ʣ -5j^ѓ2#[s0?\H8G2+hVB۱<:Zy~Pk~+vzPڱ({"j">;zx?>[_R׽7u&γьcHT/`t4TUHR}biu ZR)yRݗ|4(K*܍y ei?v_WteeY9t&T.\WW􊽼yMw"!.d:~m޹WP_Q<c׏ȼ{qCqyQ7o2;:p0ϣ'oЁ38VZ_䅲Vi8{sJ[KCmv?/K=gG3p<RM6(;lv{_> +1<Y_rj_agp^=vĮ]=s޴v͡}q<BΗ.Η޺jwb+\8JhGry(({ }gA5ͷN`Lt-"="fp>P=|a](^fcu9_{{B]֫)6iE;Me6U͔G7{$ < Agq7ԚroNe77V{7b^.mtS(-*Iz91YIY[7X}sOUB젦\:Ig7 +=T'Dotqعȃb֟tB]UJ'U, O[k(oINDڍj{K;B?|ĦChЃ6xzY=ۃ14FhLFJ]-*mE Tc)l-IR1Ő{^NW&-f-y8ZFK&Ax.u#mmnYwmϬnFdF6I*{(je'DJLNGkly˫J!C+:$\# 19kU^ل(m+WΖؤ l > +endobj +675 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 548.554475 101.433071 529.804475 ] +/BS << +/W 0 +>> +/Dest (cb121-1) +>> +endobj +676 0 obj +<< +/Filter /FlateDecode +/Length 4764 +>> +stream +x][,~_Qφu20?1fه6KaB[*2ϔaNӧ+#t>}2U**c +^o?\~HĿbuopW%mN% Br:Mc˟~u:j]TֈCQ%29vQy%p)q*v +0æ8̰ZGFaA* T0C%Ci)Y 9VǤ Щ91tIx]t$s)k%- + C5(z(g[k{6W6ڴuuEО%Kb؃YyDiLMAIJOkbYovO?\~?}ݧ?.oQk׹r}+׺\F$B"4E*j9-eڷd/K9]TW~':Jszo+VG6Z_]~i !䞬GF^ZL {kduTi댦b_*=gd')˽Uz>n~QHym8\u}Ď%zx抗e.,w=HiObW ҬiVs;eKrCΌ0Q#+e+n}l1BBMU{xcn!w›< 1dMb!ɦl?zT>fp|K'ܭ0y!m,MV +C*vG'ͷ%jL(y{֞΋g\~8Y͝gczSiTMaP牶Ly[8gGzvF uk+Jh*Fy.oq둝{}N=5dYc7ȺomHŠ +(h_Pm'_<ِGϵ o7 *vY)M/ k*oAE_/m\wg9ݐcwm}$hf{;~rݽU޺4j,ddTLuS[<@|,>W1Lmcmv BC6TlBy:]><㹗_Ρ}Y<(庞G[;6*9Zrx3lHj+xbܳ'8M|e0G]y{>$CNw<ċgu5di_:3\_l)a!/h}; u2ӡj?x}/D^/8&wyQ^qX;:p2ӡ`鑝3{;8eq2f L+۰97a}d=? q@Hݧy `]NmPvY7[Z;@w2ӡ`=K֛Nΰs)&v驝}ӆڙCg8 KNy:_'/=zjw\c{\;I+Dhs2! ~8FG{=ϊ[M[-sDs ~xklgêa//=lgy9Eu>eq:W5a~Eҡ&^5P5˗E8 5/qXEBo~6-Be :?I*s[Ջ㩬{tۏTXPz+h(+Iv wrlis7%d> -8z:L2bkqb d70biNi\=l8^Z>-}j6:ʚ3ݖ!rzKN”+LghǴPn+Cu]mGi|6ǦSC״S՚YJZTm]- F&u5e.Y̦zN]ɶtcD ZO*|7C6S41"Mep76uBo&KdfѹBIye~+F徱Z@ͥBhKEɤ'm[C yDRC.-23-Cvu1btƞ$pcVVG2YePm%[m|Ǒmb.nֈnIKVע% 79=e%##c&=mv]w6l7TlnCVT'؏ކ*m=8b쯕lWDZXCXcdPx PcPT<>[-=RP9Y _P5}7׷S)i#u|Ž +endstream +endobj +677 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 676 0 R +/Resources 4 0 R +/Annots [ 678 0 R ] +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +678 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 570.123057 101.433071 551.373057 ] +/BS << +/W 0 +>> +/Dest (cb124-1) +>> +endobj +679 0 obj +<< +/Filter /FlateDecode +/Length 4781 +>> +stream +x][,~_QφI~e` /~0,}zmxJ]"QYY}g]BOBL*|_rzI)ய?^~%z%梨ZxCI`71\/o/oA/Zu;|$pn%nzE#%0;^Z {6W63uuej"-L~0W4Xax0 ͆O"V>mTZXw%3hӏ_}鿀Oo?B73=>0ggP"-U_k;7W?[=e,e4{4ikW#(;m>iMR?_?O[uOASb\/?j9BmrWw5 ^nRa9w'֭۝ff]4\B1K6+j!Iw?s 1M1$Hކ'ZbPn[wtUή# QW3XeiJT[cwZJ@_v3kqh8V' Akod,%|ȿ tb_Ni#;Z߰C|e#u +-Mu0Oo-be'_P*mB; Z76~6v%z4)e'y˩Kњ_7kL%b߯|_|ЋBHڃugtg>{pVr +Bm 쭀llNVAF~n"rwd'YUvtzF޼ziv9SpFq5T؎O2rkB̦xyJ7s)#4z@" Zes'< Gh_BVYIAw^E]6iݦ}/w76\ȱԻ.ͤc$hfc;Սi[^~}ko)NMӐ켃: qoT{ kK]tjw%[Ys[;!0X"8{=l^~9gআu3~Wzʊ6Heľug{|.ϳ!1Cs=Y~B{w +wQ[Nš=(y搴-iXxv?|cDEg%|#W+A23^&!ݔő7D7ݓy1+#,{s.d$tH˾܎k~+3j^ѓ2#9Uz▭FF݌1G@ۈ={h;īs5y~FG`!z9Xn56wZ"Kztؼ">O2T)n3we7vfyDZ/Ž{Dg'<]g#cݫG3fLT#ǐs}LGQPU"Zgύ,e< ՖZNɓ]ʢye<ʎc0J';beН8Sm{ezjzlhJDZ"!!dt#tyNy:T:O0#謗F?0tu/GdC/yRjow7wrOO=[OO79{C~i~7luډ9t#{Qoz<φT]uQ|rLxB}hVIqcI%.'G^0C=(+?fsAc:&پM"s(ɑ +u_-A1w?FeoC. 5 ++QZ_atQy[^Ϋ"x~J)ϑ5]s/ټƞ>Liî~rV.番m@kp m|Nnűz#g9 X:(t#*EgygUɴ!~)N;7#&azYN3&<~Q8^Z-[6:]ͩȖ֊!rSբNBOWBv(X[ń~/T!6٣0t:ͺ5uj,}tr+yר.̨ܥz? ׋ F:Y)05 .t"SJ#kn Ye찖Hځe׭}Í _/8~p z\hY]ːҙ`7 &3*|/uH9jTBcNYI.]:ƎY/HH˘T{C:wsq=N Fy 07k,&V< >2&x{ىD:4^z)Ӝ9j:cGYH;]\q:[έ1/Rb M؇ hui}CCǍa|Fv5ԝAur>dbӡkeɋ=Ҿzi[:Y \cJ"qEIǔ3,#.xdaL$w+,.XAvܘfyMVZ 44o~&6HکK߹7d-a{[hwxeس=Ccns1ʤid <5٦6QtL5Y/16BRZD/S _=D{%RmђpujddR7R&uʚ!aDjd⿏5ϴ j?e'Dq 4&j5e/B8 %5p㽒H+5;gFl;~-+I9@ +4v;AƲ!/#-orY +z)|'ylo +endstream +endobj +680 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 679 0 R +/Resources 4 0 R +/Annots [ 681 0 R ] +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +681 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 573.873057 101.433071 555.123057 ] +/BS << +/W 0 +>> +/Dest (cb127-1) +>> +endobj +682 0 obj +<< +/Filter /FlateDecode +/Length 4741 +>> +stream +x][,~_1˺_lH~0Þ=gmžqZVY]vJRUSI֨ +U"=?~8}t;&>ӿyYV{bY#B A:0,Dy@SX Mq,a +B c2[V:(˱ J.0,%!R<"5)'<9VǤ ੠91xIx]x$s)k%ǃ,,X 0ރ (Lh%8J> *&a$Tƙ~ - }w1P p9, U#5*$qXw +߄p]L1rX%aFa!#GRDabxOxtpX"ayDam>`UH x]`aܕ2$ ":Q/_Woqu\>x\?ϒO6DZR'S..4nMv*|z˨Aޚŵ\mC-ꠥ)6ίy~Oo=ف4BhD,sDz|ҮMjۨt0_c4s*UR :rP]FE A1.w1E1q KֆM'ZbEW[78t*kv-)I NoǢ_6c оYi)wdh1[ݶohRIrdi=9l_CAKOdd ,/-".2^u}#-ٳSZSuw?٦6_6gV]z~FyյA=(us;b%z*se.zLU_K^izmCܮ¬,wa-8u%oAEUoSt.Gaݞ]ݥvzD˺pV\ҡ+og=飓>~~ZKZ_ƌ5UܜOћG\3n +5~㭳3'miK7c*wI[##fQ{ZzGo;J:]g==뵝6MZ9߼P#GWm[9W)n_ֳ^7jMVku5>,u䇱jRybY|4_ocy~{tcrډEa,˧ υ`-`퍞9xMoΏa*⑗6;ڏu^_`D=Q/8&wƒyΣco02_{a +<=z-;Cfҗvp0KIiiܑ_ ;8;kb7m^;c;Cp4`=`?KWyמ]:טwҊ<69<jO ++.pp^ϳi)~,lÂA"O}r8$Wܱа=^嬦#m(coE:}+X*5w37$+U˴^WDu?@W~u~O.k붃v+7Pn#GS$ +]=@kTB|ls):=Jo6ZI#7Ա =.BQ_*J&=nK4*!Uxɪy]#U4 +:5W~^uU'[e^ Zv>,NFq>_y`3\?J:*]0B'w}[:;ɸ)s8Mg(xd1R"ͷɞGI.㸖 -ty{ A2}/D%n3`Ui]6k=eR5U;]@u2fqb d>6}^nMm AVp36LC}H!#nBq;Td+1!JI~;P]FЭ<ۇٱJ˭ 6XUci%#'۽k jF}aDmSM6|$B]C,3=ӂ5zhaW7سnVڪ:P4MVZ+1T|VՒ[Y-StOhD7ZoRukђi9ɞŒ1wڞTڶn {d*{6I"{IYL\WBT/c*mi=0W"쯕lWDZX!12(f4^g0uPT4>[lnSaNnVW˼ nM雰 #HşIDIS +endstream +endobj +683 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 682 0 R +/Resources 4 0 R +/Annots [ 684 0 R ] +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +684 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 558.873057 101.433071 540.123057 ] +/BS << +/W 0 +>> +/Dest (cb130-1) +>> +endobj +685 0 obj +<< +/Filter /FlateDecode +/Length 4776 +>> +stream +x][,~_QφIw g1ه>=6Ka ns):I\%` l[1[2H\`Iѷ<-> ؂HQ0yUIZEt Z0k Z" % `t$. `H\hӰAtZh` A Xez%qfh%jIGיQH~PQ 4&$.@bI\X:bI\f Fbf )؂O₤aq蔒 ! 0.y%.HVa%0n $-!-0nIXo$-&{^oF f`^ vx%0"2MD^x]@|&+#8c@seg@F7 `&VыVJ6P_cuiܢC72aXdr &_SV4_av5o=e(6} R7Q˖ɏ]6vl6*`1AR=x(NXcu`x?AZ{jJ<_Vs7FKu;ƾ^,سѶ՟C A{ +Ė`vbf+,zl +:H&o?]@ŊuϾ]#Ә(~?>'(ZF/s\rֿ#T [˫IoSH/k9mzlSB9Om~ױ}>mW<+X0Ak1׹,_R:UuRqσ +TҸ|NPǽ ];45K R!+;Pm[_l4'COw +p )w!Ny6a}c#:{tDȈ4шj rPwraf`߾G]e^LڱűmkZrѵN'Ļo>^uRxe2rn꛽;NV/C*Sc\mضrP|Sۡ6!ڦ'ueWhY,}&*6GPx |/Gv~ʾ:O_e;ToGf3m|!^b^G[3 +\Ql^A>5Ol-mm^ngƝm>ί:J_nsnGXr~Ǎ6Wn.3c;bAyiVdlw m<1۱{cc^'+OqWӻkW{fvۏZ?: +gn|9R_o󢓏m<Z5{adĒ(1DCx^BvNYaJ||6UXi8wm [˾=.X8"%{)f${v~I}&΁żtw=dm\伽!a +:6= d:u5LRgss85>'%b߯x|_ל׋jHiĵKCtP-bP (6A'')TifI|(,(_j3soo8nFga};ю'ڸ=z|ؿ6Ҟ7Hҳ_lHwbٗ89ȽP:#^=i/3RS-nhd$]9JF['` hthqy#~׆P NQfkT)n3woǍƧi7{t(<5)6L=(X5}#f >;z8>ok^=6Vf2gúAyLEU[_K|u5di_:M@\_l';n]}ɷ3l 8y6韡w?Ɏ:er$^گnܧyMDZ"s@C+#u2ӡ8x}Gg$Jܭ{'<c_`fy =~#'C'VҷNpcg~4gwPyhݹmv7~~(~ >t/DZ6u~(x'<zg}ɝR~x"9Npn=n=upnP;s>yi~qΑQ{4̝>esZwS~B<qLȤKu]wQsK-yܨqF͎EWYnŐ!/ƂI=^_d_GK7(m ci#> +endobj +687 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 555.123057 101.433071 536.373057 ] +/BS << +/W 0 +>> +/Dest (cb133-1) +>> +endobj +688 0 obj +<< +/Filter /FlateDecode +/Length 4596 +>> +stream +x\[%~?<,R1ؐ``x&vZV׹Lg9t)U*mKdTJ9:?|U?_g)*orH3\T1{\'똳U9kw"bQ{ ,UsXGtdEA9y1$p;2İѐĂ&(j- &rV$5*lȋh%s*D2HB>I<,>k#yde\MXi Z$>;@iϧϾ/ />5}z_뵭qh68|hXh^kϣMzZ@Ez.h_deuv#eЮק?;KYl΂mcB_U-*nTvr'hs3{eNwڏLkty['HsC8nw˴nL~uW;q,jXy6U_ks6zۢtZEg=`lZh&p-zBɺC&')*7m*/+ګ~+m^ήrmea6mz-83߾\ۧ˸0}w&:؂kgMw.ͪo5kfG=!Oj~FJKSSE(C^e.X9qm$fc:g~8oF c;vb"OUdy^,c.Ұ~cArxG{o6ak}t:5>f#Sj;WoY|yrs\ثN>] ?d.hmV~<&s8ʭmnz>KOF1|Aouieo/Lc'o []ɮyL[Oq{*:ynQǨ6N^O3ᔩ~T2#<~+6~ ?[lYud^ϫ|t_Tju<1/㱓?aS[=qkўo|x]ֳ\WLl6Ƨ!gT.yܟi@vw!/{m>S|CӶ\j2>zSeH295.ȸ('?Bulg%9>8:IyLJS\tO&ɘ?Gܛ6}c(s{~Vj1^djt^ta~&>c1(o ǫ;Gwy[!qwy=SD2*Sg6u)mFaV2'Z:lW\-[ BVmJ] wfnK6֭EV?wh]TaX͗b&6ݷ j>eMzH_UxuR3>YAܤ$¿].%TDŽ:g>UtZP`Fy>GW:df{qS?tuV ir˔j[[~,nv7lqy&WE\8{22l>mpJmHs߶T-&p~s?E>:LH;$O,^ád]eR*g%^+C 8}//H(y:➷+\Zvs<ː8.e>w0]u^ ZZ(FbݑbK:s7N zgXNt5B;kMmࡱcE`0[z_W*kb5`Sm~~^rY"sG:.;#}u^'ikx>ZH|V5]srq%CҾywyu|=? } J/;:o~X<[.Ǣ <>pgW7rهqGI/`V?ZxAv8g|Y܊QK43gyFI$/[v_mFHXsK/_$Qd&֯ELkGgTy-|{:w:W0V& +Tֱ@D`dViIz(|G4 7D=u7Ȗ<߷U>LoemyxBn>T~kSkiNsPS}+ BN$?խi1;b#HVN =r5-{4!put3 !9]')nk&S|Q(Vh\o&NNzˠ7LF7 *|2/K?f'4CG4`(Lgg} t؎:lc'uӱr&&d]yGr>U3r/QiRb.S#KvQ\N!NLvȹzfS8ktG#M7ZT)/̝<=>UfS;wLcj> +endobj +690 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 367.323057 101.433071 348.573057 ] +/BS << +/W 0 +>> +/Dest (cb136-1) +>> +endobj +691 0 obj +<< +/Filter /FlateDecode +/Length 3514 +>> +stream +x\K$ I $``#0rxl9瓪jcf.zD#w$Τc-FeOGg%rH8=p|9]srÿcܑL΁?ca b^aec彉|PXb b($1$' % Kt|1\Qa9 9Y aRxΛM )SVy<I1y !l; p/@lUdYEB؝du%YlNe6>')sYo|b9aQPR + K><ْc"x 1%[x7[vC(*@P,6 AL%6$i(@)5썄5`#[%hHqoƅ5,]*ְwI`AD% îj@&EK๨aAbLkX7%]ҰbDÂ! \E"sItsNÂD٠`5,@F-xbK4,HLI5,R<а)ŶװM9* RRn& R,dIf7Bd|dLI5,g%ԛ$/K4,lQoS6s(f԰ dHQTdAҔ&:iY(9Ro& fDF RoF3Iw @V-Gg:33'[# EꙁI=4,SPO VtV=6_cr |D`13rn!{1n/?xB=g8$ P=og⩊^չ,c"^w< +Qr_7#nu d<PIW(_pG02dV6~wiR,9W}8|_}0wχ`aK_~w~{oTp |<>Oum'뵜V#^A﷙kקe]nFu+4W7.ѺYF1#N*SWQU3ƅJ&W6ܰ;JLƱnY}cβ盛/L[li-\?絏s}t4i[t6I{'hfc#dK彝[vߘ~{}޸ǣٲ4n}$zlw\ vzh+fz>ĥ=6r,?{EU)eH$DVQtPփذBQXu{7@eV+lT啍*)p]n˯RTSZ<)j]e|I׵'y~˽>B}էmSyĔ蘏dY\SF֗>^{{<_\^kU.)_\nyVrCyJ \sk;&{7v}rY}V}:q}kYX!bdpr͕\SKVdqL(s1x 89N 2IǕօA[&'u^#XZގiS%i< ;շ]!NmS 3 15C/~ +tP{14 S7^=Zx;v%k&3{ |Tz7ӛ<{Ԓ/WpF+OhZ^8&UHNnLuD'qmv/ҿnwlCl+ cKv:@X۟36! n/wWły9t|ɱkܕG滼}TS[dh7C0 }odZ=~>0G)"+oY;}9~9^8َԱuC#Yy_8I.7bwH2 4f =Mx-czsg{8.9v_!vOKa.7Иo4xYȡy@vhԗoP;TͰm:c\<$gH-dӘmt1lE6@4h}d +밶-K~JD?k%-׳% 9g%B Nѓ|mٮݢq 1̟|w1SW'!.tE&wŦ!R^W~Vr~"J敠#6tt^ȷTòUoupmsF9 B ތT,1Kқʋ:h>6'أ. +endstream +endobj +692 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 691 0 R +/Resources 4 0 R +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +693 0 obj +<< +/Filter /FlateDecode +/Length 4509 +>> +stream +x][d7~_H1o$@(BNf|sl9]}&2v*_WUn-~r3Ԝxt|&GZCc 9l'&lr>e÷8k[5yYL.9ۨ*XǤ3b bLYbP-QcA (1eYbHcA WcVXBUy\4 JO;JUe\/24^5)X4AXxDGd5dPK%`Ud%!v+UTd:W +-&sDfPyXQ,ƪ K kY?8C.EYWwiYϤ &_C I øaP/P/PA_T:Ӏ X)iHd!\YB% ,(aASа m ֢aAܒNÂ%뒆 1J"  T5,԰ tܳbn4,@jX@PSA@G\ p785, .װj`I؊X % 3\k0hX@]1sw}^zIX5B@a3a*ܟVƕ4,]W0}*qeBƥT5,]8aA>x5,]5pj5,] Qq4,3tU^5aA 4,8TÂ܅{$஍!hX<װq xXʚFjRhә3΢X@ooU ,*{9i1Y鬺m` 'E`>ddŲ D#…bؿ~{8sS6"/R4fWqm' "9{YkۮE#(btduEwZZ&'1A~: Y@F#8$1v'yIPgWS;~Oo1џeS Um<Ωvp.%JTcy IֺjJfbY7, p[2gϽ,ޝ7&4[t?XFwݦbBV7!+LӴѝ$JJ jjGm$ #uxFY9OȤޤ-XX(bkom- 9/,ȷ~f I q7[ӒF:j|򼡼fϩs=u=/uSA~Ck}HyqY)Z :?8t_~ͯ᯿BP2"Ŀ[ z׏s_Qq&(  +wﭽҫ/_n}Mkǥw. 3t zog,h1,mD+~ON_zky`W t~[U&G2Hz߭N]g4[/OBZc|TS{Z{Kk7I+:1M{w!1v6S59$ci3dzyTZ娴XlskYӪn9Nz;i{Y럄%&} ZNf~Zm߯r)1ʭU׽^8!s<￭;tHڅwiR/yr&1>nE/?$gj+ay>]B. b|"Md^ck}]sabL12_'4I o|γ|/:5bi'^ՌG␷>67߼p<qSw^ =ĖIaMV>"Ñk&2 __o#m Yi1~Uv'RRUgej7W[4njȷS!~*ЮVZ w?dm=vj$8Mқ糓P]8oyr +U^XYdTL"مctЄKAyg+uVУvȸ.>8];|K`}}w7#e`7Z[\ZNf}òEV . Lm,/"X1>_;6˃ZY^G+ M>{&~eeSǰU|f>EFMoD /{o JvgΫP_(9q,y`nLi's8ǽi,!݊,zlg'(2`"tXL{;n_JXJ=wZ޷'[ᕻ𗤧r>{~g_s=R"7#-rVo ‰Y}$s>Hg_v=GNEϿ\k<0Ѿ8rg`݇`Ɛ#.[t; Y\p3ha P|sz'~G*U!sxx׮tOZ̈vz#~ESll]s -}>vkT|-1'#Sl ק5źㅚ2ϣz+[&q2.J0۸'?.RRD3Af֎ȍNp]I8y{"h ׏/iPh,+DḶGp>b(8$Yv68+$`1n=uy5T&q.U>F<mST*"ٛvЂj%DŽИ'ߊ;#~e{ӄ|1k|B,ݐhB:rkkqTʬZ(CWh'[gZ-ӿ1׶g+7!Klj6vӓ bLj.]UX7 TAeio o$iipgY.Y/WRAwP=.- dhn0 yRvt08ڝ4IqW yG;lw2o7;)@a'>5r_v.2847C%y}Hw/<+vغ!KOj;|v'e#O+l0iV{I7 $ƩCAt B3 +d~L<>8N8R.pX&/L|M≋N.yQQ<l#&d'9"0y4ݬl(DA>ȆNFEg/<ȩh'Ic<c>mrҘm +jQvJ;N-.SZuUKy(ge(1YIgeFF99:;KX($k*͒m0h +fdjgxVY~+]+N+2K8jr9>-)CTu\KQ0͎Ҍ.6C4 +^ZuH)@ +0wvkj}6cToϕg{Nf_̀ +endstream +endobj +694 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 693 0 R +/Resources 4 0 R +/Annots [ 695 0 R 696 0 R ] +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +695 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 563.554475 101.433071 544.804475 ] +/BS << +/W 0 +>> +/Dest (cb139-1) +>> +endobj +696 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 100.307453 101.433071 81.557453 ] +/BS << +/W 0 +>> +/Dest (cb142-1) +>> +endobj +697 0 obj +<< +/Filter /FlateDecode +/Length 3420 +>> +stream +x՜[\S̳A IL&rsN_f+l4UuU"Rsλ=-%.^T)N.⏻Nؓ5Q>J咳B]JkT0#f!9We!DK33XrsUX*j̆*x{SC % zt\*ЅL٥,!KW]J)K'%V,]-1xK[Kt pbo` ̖.TCW` : +/G t!dSIJlI>VS["{t [/KNDΥtu`"l!b` &"%sK(g'ĂvRXRF'ft,-QEErMl* ` +X,`-$lA[X,`V|  paX .$ ȅ+U 2i],lA],0RwYX .3j:X::-gb̪&GxRb@].;AX .E܂],KBbŁ?] $l`- CGl9BZb "krd `L]u!cčuQv!j Tl gX;DTXԖ Tł"o . X悇뚺|-bAw%X(|IL]vp`0vył"W bAw 1u Y,@^:O w} [:p\-$pWæłU@b tȂbA*jbA:|.9\GC,S[$$֮r- 2&X eXՀ?؂LbHJXc~ jkSW bA\X;M, X,`X,@(r[[d[,@ ] +d +2X$҉}fXP$9M-`` R:LX,f;Z,MlsLl( J!ndlz؂bAil{57I,`]~X,X7ł*%"E1uIS5-&dl* jJ, f`<(q+d]db.Dfk=33[YtYFVV<3CeI*E>&gLXO#wȈ%MV, Av_(w_{f9)ka)-g,;9!ӸRa3 ~onw0{jzW4|ؿ_ֳͤmck8@ Ooo/bd}5cҳ:NZ0I݄ͮk?]ƌfdIi㱃-xb/mq HbpB+wlܥuE s`BNqz4w#T +w_x/k9ֿl2)[ߋlF9LZKlaZݥ[֠mol={8l?JW۰9o>_~zm;.r( _QL^!%3zXs!0F$'^ҦݹЙ !bW"YWN:̇rχ[ejl;WeJw. 9bh:SR5 9y!3FxsnHQ > O`sB3m44OYudg#zFzozU{_*,\q:~*{Zٖm>WS9mm+w)wc Gm|rg%d|_*_LK=3d҉|ν':^hR'L~Wچ1mu%#zuJ\6X|p:pZ*'>5tNi[Cm=>;j?n}OvP!{8ϻ<kIzS{}\xЦ),j.ߊKm_>7~b^&.Tq:@ׇ"..q!qtX[ے +vaӒΡd}}Kݍ֏>l cC) XPoWSѠeGMq`? O 1I'd}v &<-JM>ܚjKx2vil%H'CtW5-̮fMZ9r3DYnk-ʽTUkapu.} 2F475L<3ڗoLxE+qw1`f!Ovq_6TXMK<˛ ݥq5:ѵǃ4SX#$p7+t{>bGi&y.Z;{WXdV=~-a8RśA/dc{k;-HRkj$$T4}I R2<vV2ps6z[15;Z-=m.>^n?|gu9F\e֭~E5-ͺrt|@ +endstream +endobj +698 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 697 0 R +/Resources 4 0 R +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +699 0 obj +<< +/Filter /FlateDecode +/Length 4776 +>> +stream +x]K$ׯaz^`v6`tc/))233[x{#RfڣWeS%E:>>̔ZbuQo +Gk(M#|R .m.ew:k]TJt6Y䧔S2Aa ++)%ŠvKX#$rg%A%1ȹ2߳ +pɒƂ !N0E(=j +Ӱ wJaA"n&@o50]W&Srְ wKN“@&հ`<ġD.aAཆkaʃ,>; ׀U|dBw k<1hXljX\ְN1TRbCȧaA2 @ʃ,HS%kXdiXdAȠa"XDjzY`.j~@@)" RǠUݔZelҰ %[ KMNa~pI% R,p M!2x ְ.xjXdA4aAvYf4`F. Uk<2gf 2ʼnCE &4*&4,p4,@diʃ,>9;N79CgM7@D|'5,gf䛎4,_r5,( DmͤaA!<Ȃ(~ LB txB9 @^Ȝo&u +IwF p|oHtf3;[؋*{9i1Y`Yu'Ea kfN< D=bؿq{,S~6;)` +f +Az6 + ӭ,5nuעOAFZ +tl/k:\;ĵMPOb$r@tCaI"vyEPǠGWC;~O0џdCeYSUx.Ssm,{(hsɖS$fYEQWh]Ɋ1&݆l(1­ɜ>#lh6$鈚,+ e݆[rlN EՆ:u zt5?r +ʮQwȳNқ9/"K;O +,F,7{s7T'/]l/x#uկ*Wi!KQ%>cawYJ_Pշz  MD~Jo}<c c\6k(74 nߏp>f]  N\}.AsiɵƥpB"4p>7)l+Ѻ6Й7.==s*LoKYV.r-246>,mMfƖVuZV[Fj۞˚pDuMeD Cb4I3Uceq?Cn>4퐵Yg};7|@hߎY5Yب*wfel|P,V+m7nz/-WXAZp(ꤥEꖶlK/뙶h2~M+:*Q5\ѣZ{q9)χGeb;2Vn}.1n} +^V(d^Zt{n ӅD6޿*TcpdF[׶jH{HhԐl]! {*\ 9 *6zU/]^Nw~7߼ԖN-ha8‡66S,y<}EXF\$|6z/FUkQUi/i{_՞tH9v NR F#;|hYUmz]^M2gOM^ y[ e>=a1MjQs +mi7VM9> 0"Ez?.ay/͜h).oP۬H :nnjۧՐ~Ilb+?l8IJ!K~m>`r/Tft'h*%pVO hͦCᡋ>qcGȧeT˟0 /[~[/-I6ֵbJq'.7r[YZmuĪ/YU`G̥5D]C76K7'k9hvaX®㚶+}cl[Y~ӧZ\XpGS#+|[*\|Hk[rš=&.$$Bc庼g3:g"ʕkgYMeUc]\^[n}!Cͨ|XV\b*/qy5Gg/,:f}rWۜγz .~D|3YmcRv(DYĝMd3ly8k49_V +V~!˿w|h/šYFLIX\t,V$ڢѦe2v"C0:|?wC0#|hcq]-rŔk3i>&>,$~z>?.Ë֨.䓪r+[V&FzSH|XX[n7Bz,!&%3ǰOXj "y(G~h6l&[kgjl|꥖7o~7?u~-X` MzWn}JzhfC~9[H![{|4-NF;i >Lf8n'CGh~hdZ!_8wQ3=Go&MyU L8$Yv48*^%[o1Ft7ȎO +ױ#~eZӄ|1k!Q/WZ9t:BKQ4QNį>b|&}Mvv)-a` $ыv;9a/'HOpFg3TNe,u$ f'$鱣iFY.RY/D Jy^klE*t/7;xɶ[]'35tDӄ@w)C]5YCowcIN}-?6߈і̦['L_Fg8( їJnщ_>'VC]N;deE?:=[h^ǝh:r%7Fn#ͣlhq^Өn#\I C:fi|܌{mp|5?nI 9\cI>&qEǒ.9QQ\ːl%&줫'9"K?x4YtQ" <ЯvϞE<;EROzח}CC-/| g-hu!mTPSjUmpibj2ťZSKqVzL1?HM9+]AJܕvۊף% 9љQǒYEYSi}(AC8"S3]Wk|G}VԝcPMDc&xu.Ȼ 1&Y~. :C4ޏW+F} +PgKcꗉSE~7󫚒9ظINlk +endstream +endobj +700 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 699 0 R +/Resources 4 0 R +/Annots [ 701 0 R ] +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +701 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 512.554475 101.433071 493.804475 ] +/BS << +/W 0 +>> +/Dest (cb145-1) +>> +endobj +702 0 obj +<< +/Filter /FlateDecode +/Length 4689 +>> +stream +x]ێ}gy)ހ$Fag6 ϡZ*ݳkNJaUjϗ +R"x/-PIe{mrJcT0zҴVyA1XN3H0Y"^*ZϨJK`$2)+:q: QGVS^s:ctQ8lt[lYbtLduEo8-]-@W`uE:8]4@"t2ii HNauQ$-N`5Rs:[?GA"{AQG:A:DAt\ӊ##a%1Q0sYl18.0M%%|KR9 fJ GP.9.0\t1=kЮ8.0]CJr\@]cP0* kxHs\QX:qwuqw4xW[\rwjҎE$ wA!w` iqn "?fN +q\`*`X]ʣǙy^ʹqb2X.3p\XQ5 VjNU:J ,x~p +X߄.B8.]ВL c!r\X#+18$=9._-3 ++FӁwM&'@/Xs\E \>V-^B`AY' Xhbkbkx"*xlx[6> O0o1*sZfu:9.bqb)09..q\3SqS,E"hxN8ӅN) -9.: tЈ%՜d3H"&4&8.`r\&V[% B) [)D [q\M4Doj2śh$Mx̼ƥxs\ X(i̼'!;q:[t$ s) @ M oz6]޵L8@ ϖ%zg23 X[ٔbl@|S& `&"جU(d Vp>)*8x(vɏWS,(8Sii#V|0JLB,IFXT9閌ӛrUr4NV[2&NJI¿}wPg+.9rؘ0i{~~w޽̑')IJiW*W!Pee@Er;R TڋMe&^Η>??m濓te>gֵ<_-0y 4Fvƪ<W S,/%uaZ~zX;snz2>uelX^Cn\z nrnm۴K.Pݶ{ )˞A G&JDfǃudD:א܏ZݾE[2Q9h*Ws7Z5t +>Xp<F&tk?GuOuR_y|y$ABu9Z;#k f u\#h$=ͬ[̄д延چcH}skSЊa;~i?=4#Vz_W4zsAO]]sKvo +hcVgx4 wnMu&}7\qݓC>]I![Z\H=he™CScOnC { URj^i8/XG~h7[ff" }Gpᐹum;dcֿ+3q,̣?z96g=׉W|nNW=lAֵ~pv}h ǻua-jUL[=\֬|Zאn6-.5oS:GC50mԜVt2sw٨+|ǽlco3Xph3 F=E9E.Kt[0V|Du`!v}4%V.}:{ ]*nX=e?ؾ;po5(vo93xPvtݒ\ksLm>YFm;ډ{`_>`!s>MX+o ~u{{dO}~/n-_UQ/Q]9r|͂)v;.k5wbu;ϷtXۛbpme8dxJqOWgPڲ#_|umn_u/%;oo5#gJ;ll)ߑMs8D]n*ٔ17Puǝ:Cn_wn7yF]zuUNJ6Fܓ%WK*WS{ڰm /|}R;fR}!ֽ#>-;C |Bh.:CoMy/k;Ts8D~'HΤO2g5#7Xp~I፟ڴig< +y1b:GCo|OZ@y:Czԇ\~wd&.c $`!,}jv}aoW}?e8\`!'SFQbg M3e9N}gn3IM6kG bP2vl^Q[dFW/O77酭&`R' ^7<ѼQ_)YKOr-[wFhBٜ5e{TzSX:WT3p^>pZK_Yizdn*Efb\ qbD'FMQKoCr1cD'ܚ-#KPՔ +uJ3fOn4R䱖^ OLדLi[B Vhv8Ȋ*M[.05,( +[y +{Q=IEV +|-t`/$S>n)]zl!mK$Uc)VuE(,OV(8]ׅiTɜjnKU +wb_F\{K yZب'|*ֲvdRJ8Nӷ$V?N2Q靪Ao/ZЅ[v%KoF,,ӦjjhUemFCYǵnjuY=TAUѶB[]L3<:6u|q8IVᠽ3uI>7M(K:8hasX-[ٞt}ڡ0\AFCSuTٶ,)شmäړƅ=wO}6ڋ KBxLnyXsNy*mfԩQ] ZmV e慢d2J>j[+l/ST?ڞNUn^dd[8zu,d`d]TK,[Ϋ ZͮJ̝K.?S&DK,ԤPzФFns=WΈ\I{& cWۆ P}EPZ%L*Ҝ\e0srJCBEWGҫӽTljh~>~ } +endstream +endobj +703 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 702 0 R +/Resources 4 0 R +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +704 0 obj +<< +/Filter /FlateDecode +/Length 5035 +>> +stream +x]]$}_QφM#5\l`c5nK}LIȬ욞J>"PJ_~/bL始SWr. 䣻lBr& I\0TV}F ,wd]mw78 Im +eiR)F xd$.T0HZJ\0IUZ OQ0yЫ̛)е( qA4k "& )K@ 4I\tTh$Nt!Zh A Xez%q,4֒jIF*3]E 0MS2N,~2GAHM^nF b.F ͼ $]I\,qh2$.H:biH˼ތ$O $..(.p0oIXo$.&ތzypx%0$3Mx^x]& c^qF1b@U (p/Uc +Hbjbe5YP+1lh s􉇨<`3c70-}[slv'. {?/(  +$ qgǿޕ97RWK0[V /n?tYYJapO Ɗd+u}w*>hYGI.慨sp1Jۍ`mX4s@VlVڶ N,_8AM?SE>Pa<|: hӏ_O?@_K)sٙo\/XL ~toX?5y 2NJSC{C=99D%ooi^2Uלn[dZn@鷗aY=| n\S:ű%>yI+_y:@z5~v δUְ"mӾ T0w*0$Mv@|̳YCm$r4ue4LX7C +⨏p6 +eoMki"ǰFSj{??wgϤ'<Ky.'CȕMf\";9<8HC;9=׳q~>{QKl604?bC\COa-ֆ^:Ϭ8^|}.3-΀m(5<ғe WERr€ւLʕ~/޴;D\x+iGU4MRϕy:_knmf( nGu76ܯ>'<Z}6rk5!Uad! 7Uu]~5%[3+efy`O }6Ȟ,t>_G#JhgH=?CHDFGQq!5;-^[ 0kg'UFUe ǵzv!#;lqakRK,ϸ)G =?ah{Xڴ۲+1@d(`M/ֹQX7uffrlh67sgԅ띶 ^5 -ϗ?T:hQ}} {NlW犯Gfk$ZW9Vo:`oXIRá1S.twKXзe?1ewb^ϗޫ/-=QqΓӍY99{IY q^^Fne EWOy6Dѱֱn{VwF.Gq-2һY}AV#pcd? $ٸk=6Ѣo~Ow,nkCq{!~cu%{ 'h?7Um.S;֮N+ܗ&}cqroјeqn G[m%mֽ~SmyGei}3ýVs~~iޯ asVb-u< fCWZg绗ӿv샆tj>˓IWv}oc?1 ^:F%ly'\a6N[NytX\פ):! 1Pma'C^V>5Xqދ+L*}cػg ^Xܹ#daDMG[&)ƺ7)̪H0uXtKX븍E9(bSXB/krH잓5-=GrqU'2>7uiq灮lPq}28]~ӡg<҈9K/rxY<[ձ_YGP2AjB25.p.˖F*w@Fi[/@y7([Pe"HZ'06?["iYOjy_F, edy`ô@c^V6xʹR|\aCYm*^6ŗ3?u9Yyx;&1ӔʒäKNu?\7l|v>p #XTNjN4/4|U).HH&[wsQ!V SsZ7cΡc˧Tl9qnXHDIդq˕o +$W MVDMTl ĭ|Ollk^"ͩ1MLתj,S*TWs6NB"$~jՄlr6j491V Q˗j>!v-6R劽h=UPuvc|9l([? mUd1Q=ₐuPB'ZVoK㥧u ST3U| CW<7[xoCRh +=/.B,/,t[cTe,=^&FdPג3i\UUdנJz6UjyZ=7|ɡ·[7}[ 5V 6ғj֮7b i$ة|L/L~Z[{Y/-~BÁg;nTc Ӹu-Yfx29OMlӆ%bߓuy]EC0^\ub9ucPݤ<>dɦAhz$ކ^l׎M77JWpol7WV%rn!% +WK`ꚭ̀䍴 +b̍ȗ˨Y^uM+wWL7VJ?)?0 +endstream +endobj +705 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 704 0 R +/Resources 4 0 R +/Annots [ 706 0 R ] +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +706 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 307.623057 101.433071 288.873057 ] +/BS << +/W 0 +>> +/Dest (cb148-1) +>> +endobj +707 0 obj +<< +/Filter /FlateDecode +/Length 4670 +>> +stream +x]M丑ȳf6|*gʃEQ)R!*RfV7wR$"#BRҜ5~1/Q)|I5 %f{F6&?*foc8_wȋu٪5)FUJ'W1ZU0 +>* K*A&A`hRa$0GAe40VkQC4xAYB6V9B$㍤*#6o-Rtന-6E'"l,l-:h#S>'"Q)j|rNK:c6Ì (,l gQ[V9Bt bJlev;팤H`%$ + R$1*ʖ$)J$)+I4`Wrxi58 [< -+`$q$.pXr8--8-l$.p[&H\@0SK\.r,q:p%.p`]烕v q:2Z:3I\@]gtFV)i k& k#Q栓kxג ( +Eo +;[@`: ИpK$.] slLA5 '2=.[2!d ?DIhP* 4Us[b$1i3Y f Bi6_ckfX.2cZsLJk~;owz͐)?P_a، . Q a#u1OFt+Fb+3a];}*>B!"P8ZO_ *hFp6˶ RӬwNh5X/ O?~?>ZS7}p[2tː:7B;W?׿\8EhΩo_YEVO㵬z?6x)Q g*`s@!-( n+H js]Y_P3Nw|- +8;+kMs4"Z7jjאmnK+k:b$$;GTLjzD=n=ۍ("6bq>I>V@raxvڌݵ]Xۗe~6{'Ѧlu G>\n;Y^u|3Z%_׸@ܒC},֡G+henA-ams8U=f+{WCς^x?ɈOI+YȝuH~!Cތ _}O>u~b?O*ݚ3t(^]̯XbB4m"9"7|;/#~쏝|:ViO!mew9Ok_W=Yht +ʆd.Lm~qlz\u_Z#aWMms9}Yk76lQ L}WPϷPv5lݺ+O4δȸ"ɍ҈*Q7t+SWQoTnAV;{VC2ƭLk(#m~_Y&H\AWb+Zu{QrKۻzs4҆xۿ 9e?> ¹nzgáu>յ[fx&h6$.ͪ[f>~6˶qBb>&e %-s8m=y7;zڟcqp]{W4Do:"#ZAŤYph.ü]>]Ikz"}vy3ⱵX{jk> \㟛 s44 Q}"9mZ'w[<uf =0;;Qy"F髽ɩ;)/]F j>p|g!v}6'>L:GC\ܸ_ݭ 17xgȗ@X.mqQٴ6Xg"s8n<+Gf[#mK wU;[fv9 +|Tݙ{ ~VPʞs8n:fFm_CkU.]yQC:l>3蟙j}U>:GCnQn_o8Z'šֽϳ4Jkx!Vons3 w={+|=3KB!ow_áwQg>Z.V:mػUҺxˤcپ's8DmHO>Ϯ $}>3^ +ou t[χo%|sLi=IkH; y}1+x߇C?ᚯ'єm: q|͆-^>O|'D~Yp=~K\57x2xn/96G#!xutv]>;;v1=DRAe FE6Pȡnp.?=,ˆeHMT]:%˾#~_d7mcVSbn-.Cݕ!PӲ]CZ@-ewhk +Wqc6q^&iy S>D+U&-3|0 UNX0L  ҭfReLs5T#45q?+PW#pIgNߦ*^O oNиcqS6kv2,lԋ S=$vɘp2 0'}u6v|8M2iŽ~: ˜uf|Δ}Ne2um:슝v.Uئ Zb?M[z"Z45M˅~C& 83bsdizq8+[Up/iҹiI>&qTNK:dYEFqN'ff@򓴮/ +WQݤiMWp\`xdI臧> (FMՊ,ua5ǝAS{99Ԙ\ p2i> +endobj +709 0 obj +<< +/Filter /FlateDecode +/Length 5052 +>> +stream +x][ȑ~_Qφye`  g1ه56Ka2 B%uԘ=M.E%嗑! +?hIO1zEM)+%VE'wՊt] v!/v._W3vUdCPNP%ʹ)㼠z)$` dh +0Rt +f1QPEa)hU%a&I3trAPi;QJEdnptFO>&D]N#鬝| KEIG%%-N)I[ I&L Bt)6˯?b_LYJiB|i`!8}˂ʿb! |N6"n-.Il$gQ"sXJM1&w949 BHA˶vW~$[]sj۰\t uqv@;|F (l:xdTFhU7. Ŋ9C4ݸ/㟿O^/, _nb&7:laAͺk,e M kz~ K;~@)Qy +aAuʧ`K@ +[;jCˆ]Cg8 :C;?|f@etUa4BC)wqrγ;HuphrcagYqaI5J_F֭6tՆ.S`o?x9V;?vhNsvb׵OfQCT49ĽsutzE6u6djA/?>Ժ +;LR9sĹ2GVίk~C NF}tHqMmG1NˏH3m8̄9?."{xp?WyF-mePG 5g`޿_>Ylbρjb};qhDNюdCAz8?¿B3k>i:?h=th_4?㞬th~n_9U*L}u">c5w[ރwŏa&|,ȳl裃8ߑޓNܹr;H1:OO˹"9rv6q^r0΢A/O^=Y{pc8ZîEegCQamf"wq#ڴTAt(=`q5S=2R$UdAs+Ũ&C:\gChlukm]SlrA9ؓuc6rkjW+N կr1#Tu :ԔNn,UsgmΓGBfWP#oedu1&yGweGٞ~YWDiƅlvH{}-3|n̯ua=Q;Hp:O1kBO95=.m \jٕu9b}!Fm~ߕ3Kv[w4"Yޓu 7؝~ <Ψz0-qs%7{ג"]sx\ [xFt+:*X%(=Y{`;]Vxl:~3Dl7 +r)İ_a߸\5nɻ \ϻl{dCAΖ{eҽZo+c!7fϹ̩Ml(\ «ڭn7 ՓuQw:v\31jw=Fd)ٖƽ'vX=ϴQZ.xN_+G +idgCe{OzQ[A=ݏVD9jsӗqޛצvc8s.;Yw 1{aRU +߉|#f, $ٸk;6٢;ח96OSS~;dO +ȵewCͽ>ȻۗF>6< s|8>sƥ_aw6ns-׎8Gݞ=w3w{/[D٢қ_]~_ڷ˨soC"ȷ3^Sqυn='5:ꨒ_|SHoRV4{mn}J]5ۯF?<ՙ^ +}8v[}W|%4Lg# 8w/Q8W>5k8tz5s~o< ~0K|y7FEޢT0o6NDE'<ܱ`0GCoي8nO*姇=V" +PIY|{Ic<Ԥ{=N6GkcOǍoGm&v\\ZnGҥ'L*vT43OA*Nlⵓ7!B/ܥChQ 3c+)sT$c- uM)q%jbנw;]Nμfpw.;߸8M.|F8C?NبΘRNoE\gkc8cdҊ{R9D&oɼQ<`^Ǎjla|%˯'Uئ b:0^]ub%ucPݴ\4mgmb d>ul=zryqQ?Ʈ}I&q%Z2"x8dn HIs}P;`aD>]G}MWY9 V!aҼӴG0㾘 FMƥIJ,ЯۚcN֖es϶ OZcIcpSUuhtb6ZuK9(z)K=4':>*~S6[W-9N-MJmߴWv =2U۔|;s! ?%cj[^׬o YjuJ+=a8&V_-E:VKjI>Nsj *o^-WKBU/Z9AiiaϮ'_P +endstream +endobj +710 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 709 0 R +/Resources 4 0 R +/Annots [ 711 0 R ] +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +711 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 101.373057 101.433071 82.623057 ] +/BS << +/W 0 +>> +/Dest (cb151-1) +>> +endobj +712 0 obj +<< +/Filter /FlateDecode +/Length 4651 +>> +stream +x]Y$9~_QHk|/iHaA!k^>3G:#rvVMUF?pVW D;~x[CoJVE$0^hw 䯗?yu"F뻤"#|^ZF0*k:F)ȩ`i0b +fցQa+TE"N3TQ) ;Id4SV84tZ CtIYp(*"` 9-`p:[4Z'2[`tRqhEda +N)aM0I3: [kt8x3ZuF,l-|-;0D%&t0t7 h90RAGFKc` a:b"q\`4lA - lK@ tK2* r.̔0 8.0\@)r\`qb{ XX9.0]Cq\`一 '-0`GaT0x(9. .*Cq"t j)y1h8. 8nj+5դHB,,ӆ&4 ,xWEL̘[ wU@(G3,s De8.]g/0j ,xWԜtX.q  0 %]V!p\`%qfa!y ) @Bw1G xWZc8.p]Ip{Xl1As\: [fLYR)8xW*0J#M O8.^{q:| [ Xl!Oqװhqװ.DT.l|@"`2":chUNtts\= LŊR"`s\]f@a +D9-p9. R [ r\t@$K9g 9\E 0Li)YlMp\IM9KS8.h9.6S9. J'qh:)9.* qe7HěqK lpQXlq NB vtHDR8.x=$R`&Nޅg8.Mx\l/A?!E]1^]2^V]3 #0epU(d $-8a\}j1gN2bXsdf%ٸ>/|{ߗ_ +eOARa#V8}`{ҦJ}F)Ȗt]?foSLz4;l__ƵMFhr[F~/׿pׯ }KM.0}Zxl._R_m:19:S573=0Ĵ@h MVwfyK[OMSCc$$-ڝrcSCel:Dе Fr+ujOzNi^&S9!r3Tf-jeFbDcGY5xve6xbjiЄSD 篫pkL7b:nC({8Y|%Vtag,{k=Moj͘ps&] ;!mb{iaԋ9 ٰpUlfk0odVɜ7hws;g;[(}@fک=G51]G^^+օt;cyN`zO2Y9qBVw)9ttZs?:9:&3Yr~3cdjc}ƿ!9h2 MfWL3Si >Sۚ4y2gY2G7%'7\{lHm_uL3[|c&FBs\顊giVƕO#yN_>Xw(dƳEWgY%TZDtf۾{kp;!ԣ ++7Y}/&ly,ㇾAcGA| Ӗ߯ҿeMyΆ4YӜCꌥ~g5tO:imy,;މWlsj!-yNYO+׸ǃyNծCEv(3pn(0v[X'xun֔V!3M=7eQ-+٦}MtVu y(IwQ˧G{;l(s>ߡmz3t_tL}ݵTQos]=h9[{}},л#nxw!<;D8[kh`\>0>bYWROLO Դӭ +:>iȮf+?}eǔtڃyNfp뉀%;u,Krh&y4=tֿ3SkM[Oy/ ]3&EV;e\Kǻ_s-,U/b!_f1*皷ڏ;?wK0ZaPc;kw;gG jzOxylŻkk6z7F3;>[z +wuQsٮFm~W:%b\y먔_:lUrvɰl"K/$%K'+-z*3d$;P[<*߻"ӆ&\5}z?m7ݓF UkUWu~8r`$7 w^Ûݘ>i8ua֍ +o72t9|Vnl>Exqj?H&oD Dkگ"HC*wV]Wr2XRM;5ʴySڸcƱ98tIO~UM5wv#FL$AqN?e)TNXd7OiX +OIj&9[dNd3Btj)Ng/ig t)e!RX3.5e{Tɦ4bztV; 5rQu,2攥C#]W2M̢oz-*3 +Ct1cD n*͖%UYcnjJ FԧV1C~h.@봙uVb*~XϤ+k)x104,ɑibᇢN=YFj宰q-״Ms) fRA2ԥfҦټDRHBEJˉ%i!<^jC7?,s#ʃP)4c'Zh# 5۔R*E d}Mr gVŧҿ-m^)h^q<oȶpdMY|\H4YuڜUC\Lef)G* +!*m0QFkLNpBiTOY;ZrE$5;#VLx"+^lQt_0KA; fkÂP9)TqTT*=h|2#y +endstream +endobj +713 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 712 0 R +/Resources 4 0 R +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +714 0 obj +<< +/Filter /FlateDecode +/Length 4919 +>> +stream +x][,~_QφI~cb56gaEVJTF^8]"OBʒ*|_rzH)ஷ/?]¿1bTZ8xCI`71\..y 9o?!T*~ p9K*3n0%p$A`uQۥ* T0CgVCD QY#=H:ɢ.Bt!:C'I`JK:[vNI:l.uŤh%]-]-@WuE%]9QS)(?dtZش# +3%(ElVIgaÏOxSFmՒcu/1 E QF@$qqp8LK.I,`R\VBX3Z"`wAbauf'q5 $.g%.`[H\`ANe-q: 3kSpXP.%.\ t1ܣk}0XЮu)H\`i%qZwVQ02W0H\ &%%qk"QA+q[fnH\ƙ q2 wA!^v` V+q +  3wL-d%qo qA+@WA$.]ВD@?T%.]孕 w#`MF5a +c BҁwFK:R1K\4mJ\" L!!$.YAOԅ^₨ ,Q[\tIh`5Fh` \  YpSL6Ъ.!EеK(qAnT%.ZjHr$.i'qA̰CELEC2%-p%. $[ J\L*3( 0Cc-tN:3I\\U+M0- ?`&9X$.Hr3I\l1H\lz3J\l'qAVa 7Y8tJI\uz8 0.i$.Va%0n ތd[<\Q[4ea܂;I[Lvh$.&Va8r,hm@  RY|hdcb嗿÷~o叿R@֯7z5]ߦ0[f h=}j9p .uLZ~g1Yf^ cXySYQä8-rҼME;K_鷗4އ%LLcYރ/Q,m wC-VVwa~ ]ob;A䯶=ס½ZO6t_Oҽܯ%)믢ߦr_X>u2X<{`ɖ 'pGu,%=Ǭpq|f6az\ʌes6DmPLRkfٿ*iPu,Pyy0FκWW);+Wb+.YWt׽1x'.;qgIub +smE=W*h +Z/͐U2qdE!эG"sݓ:D,}V3Y$MFk>/B&R9RN#⌏n%5[AdʬHT8:gm_ewOD9m{WP}=Ͱ][WPz9,;> ݵ(?w8tclgGhN{3~зSfss:>Zr&=)>?/qPgtS/w$/Pv¼ԕK,> +syN_;Vhٞ{~zNsZoSjϮ<[1Ԛ +FeW2b>ֳgB:.⹜cu|_hCWbc4ueȘ~"{00}/W¨;}]/]bŻ,y~&W51hj^S3ߟuvNlWיִ7;] +Voڿs}µXcحc7."`FOq<RP؎ +$;H]QʸO0a0q]p\t& AƯݗn#B̉ +QqPw5H26gJcY籜T*S}5O!xS̞ {XtO0TH6t5"1_ӥذs_Nq$7!rݫAOb:7 v +O +lw*&Kb5qM'զ MXMMY&.7QJ烖JUeԗC\'AtoT)ڡ3v$WN.b4_{*Teͥ) 6ՆdcMHG1l8*1ܘv&~(+/Bʭ^L C*NJyUH9sUBc塲Z@Ҳ1V +LHO[SIY1y<"Jsb>+P#P] +巙PKeA3Znʝo\kk:4[e/8^LO&6v2&l(.-6Ȥve&C^?ˡkyܨrLg2:1= EX huji}ACǍa|ZvՈԕAuromĦBhZoo]+VG}&-7j,W\mJ"qEI۔3,#.xdfL$6=xWX]07.mӹ6l .`yyf`> i*}zc$'KxuαW(P-˲gӆFs-cIS6%RWdJDQ2UdXl +S,'b)/=ڈJXrˌW%[ ݩIJ]KilY]f+ 3U-_|;<]AJU~HNRLNGk5B8:<> +endobj +716 0 obj +<< +/Filter /FlateDecode +/Length 4726 +>> +stream +x]Yȍ~_Fqƀw/`؇.F>ߏ$1%eхJq0dHW4Stt~jEar:?1)$g]r,v/.YW3v׿~b)X ,rn +8/0Ĕ$p mYbPNbA 2& +1,m1L$ N., 6J<xMJd `$xi޻(si"% $,[% JY ^,( JKd'"Kg\VI<'۴"+ +3+DdqVg! G' 1 +j 1DdK` SJ‚#MC`]r%,HV!D'd=A‚D!DyAS-0 &IybIX|3JX0&"zqK rK@&Ed%,MDRn. >3((@o *{ +9i1YF/mh ӸEa>3d² D-b_xG8SV4ߒ.oՔ#d ;  RR˦ W w^ Yf`&"UИ+EYeE۲_Yȃ‡Yw~8~{'pʼnž>\v뗃"yGM~~zÿiH('4__VS\ƯeC7zC/eEEim1|溙D76rE. +'ToRoNi1ZC,+U: ]yMQy+*.Yױ嗕cJ:(^04zjY>+V8t{M\nǬ>n8x8Bc: +XK45 x' gHT6!^gմKm]X$"X?|!b,qjlUbyv4Q%C]uXUy~Ec2D\EzTi9r^稶Sr3ƲcCYG +iJ58 5;|Tѡ6-.MwhMFcݷ&E!BRnz18t ZA!ߔM|ci=K#h&Ƽr|N +5-m .}9հV +*^Uqͮԩa}_ilKg=wtw}gqzwZNzuzbĝh!"?k9p 뛩kM'Qz6|IpHW>2*KNΊ;57 J/EP|N2q(E'X|vI L,Ukt;n0uI$(|T Gh`ARG DzKѫI2 @lM>ԡ{4 dm@t[dCU͇6Nlײ& ӹG]Э]SD*=^5 TjTWK 6Ovߩ4a25>u&jHʥZN .bҚk3Zث.J7um8- kDqIS\8؍oGm%vX\5hv9n/o͟4+gpȓ?hj&G F^*i:$i@Seu͎Q t[馱uP@ cI!aHTUtɮۂtm::z0wd߰m$6#iv[vl&I'ռ]l v +a-jm߱R?E&yOuOlVc2/tm=uF50 [ѓēdrJlf `ue]ǍzuՁ AuDu8M3]CϝK['PIU*\e<]uhW\tp q16ˡ%[9vT8.0/mQwcӕG1AlNjaci! g{6RO|闗|S]]C-;=ۂ4zhAFjJ7vJ:Jk-*/ժZ򈣨bNӨi7JSgorhɶȍT/b(XQ훖}Wo{A]#SիMw{֎xȳT pԤ56C$PFW{%cOUA"CS jICzLVPcjSE::Aoo Jˇr_l 3 A +endstream +endobj +717 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 716 0 R +/Resources 4 0 R +/Annots [ 718 0 R ] +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +718 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 612.273057 101.433071 593.523057 ] +/BS << +/W 0 +>> +/Dest (cb154-1) +>> +endobj +719 0 obj +<< +/Filter /FlateDecode +/Length 4676 +>> +stream +x]ێ}gyH8 LF0`~%"$ggӪ"yxXU^ }pV/NO9o(1;?י1#) +c.ú/NgBRo?GVy!T*.29'%TV0b +f^J3Pyj +0C:)8B%8sRCJ:aE[J'>@tIJN #(]`Z)-&pA l1BkN`VQ: [U面tHN t]ntA1֤.@ǽKtB0l*RNIb %SEҁ'rUѭ)&uO1;KQFDqt 5E+ HvNQ4 aG:)("Pm)&Ph\R[T( [P`Ԋ@  +ty(R\@0S\[MqJUN HQ\XIq*- +hRI` Lqؑ)wh.ʀCKq¤ ޕk 4xW:L1AEqJ[fn4\R\RKKq?CKq{褢'cn +?bLR\`£H]`¡ʼnqVf@qbR.S\gQ%+fjJ2p x~p`ߘ. ).0]В0LB cR\`ˍRX.p{HlQ^R\: [bLYe\)(x 0rM ). ^z͂s8tpHe868e:آ5 *x -JJ -p(.GMp! [)R\&c>hMq3ZYɼ]:p8bAqb)1)..Q\3` +)-p). x [ R\Gi9K)1xmYt).0A $u4Aq'ۆ?6:آ{MOqErIqMGqA[d @qNqA`\xS\GAo: -. +-p1nI7N4oz f ބGO|7.ft]x ބgKk3ZcɌ̭dʀ{1[92gA1F"0d*i?0fLM >Z)6~&/9=}1dҏ00 d@wx,:ZA a-u.#hqJE#1凙0*_>`/!4+f[2*NJgpP'+h>|`XKs?~͇FN?scR_cQ.LbT.k>W۩\s kWϛΓykOu׍Ǥ.~V)q;e4H}N05 9Y*`5tJ)ߏ߰lFe + 4$xfCXf05[عOshQnxʞtŎ _UA,5;Kpf1;@Yଳрe i E%/JO^\oǍ6A猒k  +ROI:2ordLg&>^62zuۦA1˼^JH#FN"yfaʳձj>˩NX%#][`jJY[`睍4WX9lhs84n;wỗ@(hk]ڨ|\:[HmG7v{\Z z5z`Qf9!C=*ޓ3LȺT\n~*\oux1:gvY6&nuFsӫOML|(sZ.4GCc{-c]Ac3 +2Wph3_ F2".5uq2XջV[*"Dv9bw|{ckb۷+FŬdB ++R86$DwB׃X/.Mu9Զ1QgRG.1"38(l9׾=b\AiPgáwG~w7A4dwPw=|{?;:s8:n{W ;֟K\ڠ gn>ѹȦ=fhjGs45Oz,!ZO<Ҷ~BF UG=z-#Qse;]We+ to⣵.ذ޾:Cﶏk2Ƕ9Z.gO ػUԼGCf\oz]S=5u<-@һ)VP\;;h[Ξ97ﱌ# +;o#O_QvQγvMwï_ѐi_U]WEІw<6\5ME|]U(vWp53Rm YE_i|:k[|@MQ7)[fɎ񔌺:)JztǪ.kP5bk'[':~C<A⫯%U_]vEhhӋTaO!1+ǭgBƍgVυssE&148KM;_i޷::IbGwtPϧ۪&Z3)#"̲ 8ױЖTc'W&}&MoR 橐/LQӊ8=gbk=SӑͦlPe K]PW+b="pa6n:Qu*+]՝[ U}2өRJe2*dI%sOr"_tjnc' \.rr"Mg3\.KՐZR|nxB6$uq jp>6@k+ȅr2:"~>G6+IЫ:o{^Rq C3^M*LF>![+G"qlfjͬڕUs 2f#ҹfҪ* ɼHROE}00uwO&44S{>|cQ  B\{*;9ehBjVR_ML}Qd"d<]NR wVr٧(_6uF4uEx4eoIepvx)}IOl];Ku)lym=vmmb-)/̢Mw4YT2!—:1QkTo-8DMUj:۹eF$J+;V; EWTzv:Nu @ +E&H154P>t1T4a{|w\]4'S +endstream +endobj +720 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 719 0 R +/Resources 4 0 R +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +721 0 obj +<< +/Filter /FlateDecode +/Length 4972 +>> +stream +x]ێ$q}g ۀ =C0?lKz fdF^w]fݕA&3K+9=cpOoj^_I(?'V.N^'8M ~ :㶪3\N^R gLT fXݪ20Q;I3tQPi;QJ5N)Ӄ3z +),"t:Ig^K<|tk甤-Φ(l*Q[LVEb;Y%t%Q[TPZe;u:bO*IM+00H! +:[Yot8t ocJl54ZybLt`%0$ + $01N.'Ivr%LʓJ4`\Q4tFKD`l.HL`5lq$. l ,֩%.[tpaqm +N kX.{uF ڵ. ,x:$.p]kSXdu8*J\FF ɤ$.p]wM uq8h+#qgQb4x9- X 3V6%xpœ[M: quy=.[Cx.wAIpj$.]m%xW$.] ^a$.tE]TNI- '$.Nd<)ġ.{ wVUa-6 @i+i2` +IU#,J,qA۔ D$q  BrS1H\! QAAY$.@`=ek  "F-.` "lA(qlJ9 XW%]0SkQQW%.ZjJ' +$.i'qA̰SE)DN ҅Nk -%.H&aXK%SL$& +%.4JL)` hB-X`e-; R2L[2ތd[I\U(L4Rd4NKZo& UXob'l7EMY$7N.9Zo& f`^Hތvxg0"2 MD^x}D|&+8c@U2P^x((/Me +*Z +xm8a}[Tp)R#3Ct=I6/~o/);eFPR;W&c=o[آL1 )1Us7Km62 ݷVT2 +bpO)a!/m}sS#b|ol+#Vo$7p[rcbzAg?v{&`dث>hF4 MDai\?3Y>`<"€46E}[%ݧtbާo|tnß~ӿ^n'Y58w>]\>r]Ҿ,2<˽]:AL2N$qkک_!kL=}?ϟ++aZo/ќ 75 "s0P-=ڮmYjc([bӍI*+8X伾u:ZiB9'Ô,UafKΝbg*VGHm#v~mmf(tˬ\!+}$'l9ch'Um;EҶ ־8Fʈ޿NRC+i3 izϖ^#Ua1'tsev1Oiҋ_K.m9 wC#;RڳC03 ^P^Q]{vJbc^RvMPouwHq᫷Cj샆lk> ul붍 +ft.q)X%;>޴A~3cRoQ|_pd[pmf)΍1A?[CU&i陯KyuCiycyӀ`7neŅ[Aح0Iez`fR ʦ%'SQ(ѿv<_hYn1fR+s |iYzz r-'KJ-:MK wChM fUrp{Xqc)r~- eApl:=Yғ;$& ?$ydֹ.#SZxgHi-y~V}z!1a?̌6q-lhu{5=&%==WQ&]6Mπy)˓Ҏqy>.Vd#5KHy +aX}zbm~1ϓzcѨ_M:v +\o 4nwqg2T-LC[< >xdc:f΃ԛ6~_;6\L.'ŋOj9޶i7*'P9Prƛy-ӡ3t,M<">xz+qAhx|r@h.~t 3\;(Ի_t;[x_lX8\=ufBթź`{rp''^X@t~s]9 i,w,z׶/2($'7eUX{](pޱ"0rg2/6+/nXeBXMػ[1CNl~?(vR[T)v૕_j1{V)m ϷP-UH6&z2;2)t&]:VwrmZA˝ŞeŒ2?4Ulߺl u b1 ņdfI-U74ކTMd!rSդ1}sAr)t8 {bmPظj7ݶ ]L]״XɚK Q*J}鵨ڻF5eF&5eI=n;Uv̦gFoɵtDn)ͷ-UYsnJMكU,l([?tJ 7f#|ܼk3_o> +} ++вr8.{'L*wv*|5H{wUf;Cc'veݥclԭVReLs5HZꚘ(OJ.TTžAu%v6Uj[]wrql#%V.|>csj%]lTv2&솁kة=d'qފ3:yQ?Khg?nTc Ӹu%枌A&]7Ta6Lh.֩ ͏{MBMі?>^l̷Mw󥭍j.xspv|gE⊒v3,#.xdnL$u=PX]0.m׹fyUVZ 44~q_LuQ{rikIR[.[%{eܳ-Snu1ʤg[x^UQ4LUY+1V’vKiq{)K{QiMZ*z-y[zK&Ax&u=mn]ww =2U=ߤ|gy])U_vBtmv:Zc78їDțlwD^ 19kU Ps+~ߢ-+Ioa+Ӝ/*{eC]FZ*wY +z}OTZ~j_f7e +endstream +endobj +722 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 721 0 R +/Resources 4 0 R +/Annots [ 723 0 R ] +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +723 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 401.373057 101.433071 382.623057 ] +/BS << +/W 0 +>> +/Dest (cb157-1) +>> +endobj +724 0 obj +<< +/Filter /FlateDecode +/Length 4691 +>> +stream +x]ێ}gy)ހ $Aagc# `!CxUj]ffwۭ**J}_ %B嗋JJL6x{U*`x]/l?_:EN+{Ǥ"#|^ZF0*k:F!ȩ`i0b +fցQa+3WE"N3TQ) ;Id4SV8ttZ CtIYp(*"` 9-`p:[4z'2[`tRqhEda +N)aM00H3: [kt8x3[uF,l-|-;0D%&t0t7 h90RAGFKc` a:b"q\`4lA - lK@ tK2* r.̔0 8.0\@)r\`qb{ XX9.0]Cq\`一Ơ&-0`GaT0x(9. .*Cq"t j4xW[Jqw5iq"?ˇq=tp\ƄFXqiP!: ,xW By83oB99.]8*q>x9.]e9xW(9.] A( %]V!p\`%qL c!r\pޕ xWV[L ӖV` +Nޕ +#Rq&qI0P +@"z8.pQB9mqXP-"8.9. d +7 RL&B$[| Y]Ny 'W8.@.%&= x% `t8.`p\W$FdlAq"] h؂h゠L'9.F.t$S8.D +9.4A du4q'%ۖ$?H6:b,q\Nf آL J؂(Ha2q |S) D#o"cm4.回H"DauASd-8 &؉9آ#q\KfaMD@Hg 68x7rZWqv@z-|+d V]3 ߔ#0ep*Jxfp[}j3O1-}9j1P>O^a,)CiB|i#"V8|0J B,G$#|F*薌KaM}iamxgW48֡: RҤ.hm l6_"&m>^o|ѻ? T߾-hێߣm7n DSYPӴ=仒ʷS r4n8}lKhWPlMRG5(T\w|ajK}?ﾻCTz O +fUy fHq@u}wg0d{߾'CJ"TvN?ǴJ?FLa ȔѧA=Zd|NPQ!oB񆆽i۷r>?f*S<0έyuN4}i^lavߙt(^tWw}5_5Dns:D; qf6t2 QXXt3<]^7Qm! Lm1vkzо\q>ڷy_rkxoXl5+1hrCJBso R9:CXswcIJѯx?qY07nq5_g5oK4?^hSЊ¶9VҞu7]uY=Hֻ!\jʆh6M^nȆԬ !9dž0{rSLqҹނ<1w^G^sv%}iλ'C[m;>.k'~Ҿ3acf QǤC JD>^n40cuZ_JM2-fc,0/*HVF9U|Io$FL$Fq?eiT.XdW/b_X!ǟ4nlm>Tjm)$s!3/cFNP843nbufkԖ ֊bed#slvWŚ=* tbzu\Vf8 /حϬ4O17{W"3QHm.Y81 #R YͨR݄!J1 [nl͖TUcjJ GԧW57sDs*viXFτKe&Vz\!m^+4MEdKd S3;}7Eg@*n՝YNj宰q=ײMw) fBĔO[JH.k>fIUA]' +SlvWŶ@y"v2U3nϭؗW%kn,|wRZoWwa *4vXMuطM1f*lPuk$Rҡ%TF(6jv'Ij60"mfۦl8B`n jw'0F^/dg@j/VZ,I R3lf96=T23%ȩS4h1HC6N)&;͎d2J>j[#l/STmOvTJ'vTܬlj7{-ξ%K,kk)Ւ&龳6AW#SYYCwdt\7P_GMm1;9p-{ +:J+"yC1+mg(⺢iy|#&_-JuTiNV0 T6}Jm^,CHRKC:nd ;\ +endstream +endobj +725 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 724 0 R +/Resources 4 0 R +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +726 0 obj +<< +/Filter /FlateDecode +/Length 4926 +>> +stream +x]],}_QφI1xp}]3mO(SR22nߙ[ "U)~tQS +JEQ _rq:?)fob]pɋ]uL9?)T*~ p9K*3n0%p$A`uQۭ* T0CgVCD QY#负$SHdQH:kZ)sg%-^;$lq6EI` V:bR.dA` ЕDlQAiIsrNeTi3O>Y$֓6Ì ( l gQ[UFY#Ɣjhv[eŘDlK `lQ1H4/I`b\6N"K 'hd?Vi茖*]j0sElI\` lA -Y ,-XЭSYK\`N qL%q (e ,8ׂ;%. ]L(qZk] +XuZI\֢-`d%.p]Uw1 8 +III\&)'q%@ sJ\k28q&H\L%F]P=tJ\&D˟0VN9H\:aD]tDžy /O:,qbx.3I\ m6eʦ0S4A;U>`1B26Z7T(*ObŜo|KO?^~?ݧ`~z_r?|ƫOk~]_82 CPVzn}ㆲKR*y+{lvhzګpg~ +ێyR&!Z\/׀cwXٮ!26^ͯآ8HayKm*iP=YWGZރZcחBpsfs!|Q){ö<\CϿ!==! =?<Y5ɣD\SߏHFWp٭'ٯcwBR읳Jr6{!~e1/и.A~x{\/Za}ٔ9\cQBhuγ!ל_+4UFAnc$ifDX#׭אNX CMwNy:Īرz2N|9J\-v9a׊5bp`+ׅ{{;>Ϟt=g}y$èeĸ'<Ǹ_*QMGۈ]·q0 k*kyuŏq&|,>g=YgF{O:7;sA= Ǥ'w\'<,DG +5@l賗͆qzY|zy:O1f]KC_N}6Tyfamf"v= 8k|dC +W33\c p(͸JAf2pe~m<Ο zArzy:$ߏq<zʑqamkR˞n9ևbTцfcn뮘FĴ {γadyȰʨc?I.1q{;-] [xFt+:*X%(>e>=Y{,vmt;Xg1'b2Mj7v+K;%z'ykwmywz:Oʃ-ej\^QNs{*s*:f4[_45J6†j6ƾvknfȯt:yͮcsw5F_.<ֈ,%Ҹ!Fr^dŻTgYi_ %&}w\۱y`xAI:S5Ie_ݿ;j,njsog+;hwRycȇYFw>L!r +{;m];fs(8KPCgv۫+6@1?y]8yŇGm'X:ܶSP}ّҞ I?3`@I! mxm}]ԅŗ=Q:0lƼv-ʰn}Dqu}Cd&5aE3zRf]W`6:?߫]Cnpgl_wd6~:Fpk ÎQ=j}_ 2K|n9$)rHj6^ +rC|` y tJr?&8X΃ata;Wohl|EZG]*GΆ sTۇ){TLzҁÞ)wdS,-ѡ7-+=M*&cM fUrXz藰~c*rPvh1XmLUdmPArg:vxrrԍr<8YI6뒿u3_c.~z0DՐs]By 9mKV,IZ!A%<{A"hdAL-OUyd} {;=ږ}jS'8*\3# 0*ґ"/kkm+/e‚y?oN*eS$L2 , =-Z|Cگ4gLOoaQB"'1_᤬utq,߲S+EC,~: #XTNjD;I*b,~]KR v$\ު֖-CQe7@X\=A z7TjtxS-Ƿ B z+W^E3iUhP؄A{C VXw]l\Gu u -k"{Q&J-jSջF5eF&dI.Uv̦gFTZ\ E,HV*zҡnSs T_obsDv-LUknFxxfb]|JWqe6qR/]SufLIy~;k'qyh$h{Y.c,NJ(OJ7eLs5HZꚘ(q%jbߠw;}^΂fp.;߸8t}0_QLO.6v3&솁[ةaLZqo]2MP…GY(]X q[ƭ+1_ I1}XiÄ1bzY_qc_^]ub%ucPݴ\4mgmb dtl=_zsUසu/W$h_ҡuUo-[I4ד #v,o]"- ;;+{^_<ɡ6jo7.m7FFxԖ˰lk- :yZ[=ۂ>5zhVLz睺*[uZF*k-jSXU;Nb)oӞJs6eݺhɶs8~wlddnRK*}nuCBLU7%G\g:HOɄԿe [8U!*R#s?W͈\ 19kU oPck~=\-+!iNm2A;K$e"}婥77EwЧ?'C-d +endstream +endobj +727 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 726 0 R +/Resources 4 0 R +/Annots [ 728 0 R ] +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +728 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 176.373057 101.433071 157.623057 ] +/BS << +/W 0 +>> +/Dest (cb160-1) +>> +endobj +729 0 obj +<< +/Filter /FlateDecode +/Length 4690 +>> +stream +x]ێ丑}ϯgC16腱0PU=ecQxP/Bz]N)yxAIIuN@JOoRx+oH1;?]Iª肽 vi_{Gݵie1QE2ZὶQ9%p1r*a=iZ<̠,u`Tf,UfhS gTJdu0D8V…#)91yRVq(s6p:-6J,lHr:B&xN`qF:آ7։.+:"T.ac bu:|X괴c$SJX`Ӱ:(tN'mdu9-?NƇVFFq: [` :c` x#@c G:ZA.zTiŒ(BH [c`&qqݒ% 3%k# (Ppwr\`@5i hPp!%9. 1` qQ 5Jz a9.@ +q\一:H⸀c +`u^`  -.Jqw5iq"?˻q{  U3VB8.]}. +̸E+ \w1Q ,x8. ̨ U35*% r\8oBI@E!cX.hIs\YX&au^HC9.p]Lޕ xWV[LÖpV` +Nޕ +=Rqi@ `9.pDq\ࢄ.s/!,V[pE41p\5l1Zs\5lq<o6<- lqLHqZ9-]xOSEr8.)G؂q"4V_qHUa6hdԼ>:q)`I6\TN؝OOQM=.''~eEF}|'֓>>O4SSN?:ht:tw4}6i|2QH[1:#Ĥ^@u0EPUm*`EN rO D[7~Q:;SNgD7еS{jc:e&oo?^+qӇwn\!^$vxk@hO~fy ZDsS&+3=sƏ]KW$FM_`ˡiWL3Si >Sۚ4y2gY2G7%'7\Z^D=ts9twDG1Vp6=td`ˡq.yBp塚|A6V`mms5Ԛ55Ԯ!KܖVrnc1uy #Fӕ|A3+JVGtt0зҼՇ_˥ջ18d%6vWJdbAh=rhU-\_f'+?ggK.@d)09cyHC?_` Cތ㈆Y$}>N?rHiҡxuRG͚dz\Ѵ{7r>W; +79{9g [ͶF˼}" rM)i;q7GA\;J݌csѿwGY;6]E:r(<>#: >G;r(Y|؝fd7{\ z\k;y~r^6`!W'5$pQ3=Vk7/lo]ɷwTђQiKy-%xRZaڎ[UfFfw$;]0՞pP{ +ȴ w $Fozߠ T=ǩca 7O[$HrҺ35LS}Orܺ ~ʷSݍ=4mPkXt77l9mǤCӾwG"=ms!I +9\. ^e-3tc{2yn*Hb܌Q'p;-mIH$e)TNXd/Oia,-Cmդ6gK̰̉MX1#DNtRI64 Lgb]Z ik5XR(XSG Jӹ*,MiosKrCҴb.*Efbڜt~bD/7FE:^KQD%]ˍM7fKO*Ԭ1|5B݄lhI&.@kf*~YϤ+k)xҴ2ahfY'?R-<ÁW=YFj宰q-6VP )`m/$>nI]Zl&mK$Uc.VuE(,OV(mL\e2RnoؗWiobo)| +b-lW+k0BW}yvJE:ȸ0icuā3*mzYHG|F?彩y\pLdicpP62lXf4y<>d)ĞS h[m؈uVi!|Z82:a&_1iWlNi:S{S$R:CKɈP6jȶ~j¤ƍHNmY6\E.iOڂIƙ=wd# I [+-xN9fA> +endobj +731 0 obj +<< +/Filter /FlateDecode +/Length 4759 +>> +stream +x]]q}_VG  $?,p#0pwF0`J"YJ-uZ7̝V?JaHU;"!<]o?_~!8cŨ]08|tן!$g]_wɓ].RF~*C!('TJ[Aq^Py=ST02VA):I3Ș("̰]0A: (%`"Fi7x4tF>&D]N#| KEIG%%-N)I[ 'R3R[6_w]XXY|@pɃ"f@Gm=d2;r+-@ba2oї˟_}q61,3sܩ>44mrŴ\9O͎RwKOe2tL|=׽/Wlw~?~Yd'XgG +U总:= 8azN7]i0v[hp+ˮMCl;~ꆈɱ{)׬ +?H]i=] }/ck4E s6ctK$jƵŽ8ַDdt-rS|Yok~}auig,TG^-rlKᄜ'NR~}29yΆ/%=Ǭc?"s|f6cz\ʌeu0u߻ɨxg}oBPu,Pu{ýa|3u9RvW,i+.YWt׽1x'(q9'vίM&v ({wލh̗fH*5t (g? }ueG6B qykBV龲tkm>93(G>>R9RwVgQ8^lͿs<\ 7s:D#QG␿ھɸs: <aش3l>|Ƴ \F/5?4-'RmoM~0K;}BwK-J6:ݓh;s:4t4 b;$zW:sYwX cI>_oߍԧ'bI +r_&=st#P1)^ޑ#I,oxIsH$K]ĪpOss6_;Vhř)'9ַ:)7]ٵn܋ִ|^ظ|OB@;=}dv1 SGZENg?9i#y}OY9vknkLUe_=E $YK;VѢ;K~O;3֊o߀b +$;H]w2kg +̯г +Q +[3퟾?K4i6K*DqYXȪU&$ibsN߱XN,K ?ä_溙=c݃+_.~|[ߘ|ɯԻzr k~CCTt&=6n_DKM5PWUL/.ds_ 3iLS8ro9ĄAqU-tIXe7poRҺ=:­++w-?Ȏp?U,8߽,⋃|5b3@i[-X[ńn-T&6T-|Vyi% ; oTk{(o+E5,3*U+)kP'AtT)ЙUz-Ge$RI'n1\=jز͔uk fi/ecDq)J_pcVQMvqAHW(WDž[Q54i2fP'pUgB*I]Jجe#qCR h +--k.B,o/tkb )k2&IR+8SXiʕ\]Btu2]-}7.u~-P0_qb^K%MlTZeL )4qѷQ\FkeumI38Lou~ ]X^2FU0[b>wƃ"Æ 1ob[ZWPqc_Sˮ2Z.t TmY˸;[׊Ցyoߴ {P/)V.c6%_YI۔-P\ +Slaf$6}xW\`nDMGmEx, \`/xyia`>JUƥƲ,_^9^EjY=[64kA Q&-ۜJdjlY;b+S4_yK;+J-3^lushwZKFAy*ٺnZ]f+ j晪ͭR>ό)OuR1c{&j;[^E!_(yJ+5;V mEWwْoBA*4&z mzR1f7иKz[OƟv"O +endstream +endobj +732 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 731 0 R +/Resources 4 0 R +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +733 0 obj +<< +/Filter /FlateDecode +/Length 4737 +>> +stream +x]ێ$q}gK+6 C ݵ^+?}dfTfV̪$y=]Kd0U2W KdTJ9~|*z⢨FST'b6.?:jUgVS1ŨPi*F냠 +FsT0GUEA9yI3$p۪2̰ѐ&(j-`&rV$5*lЙh%s*D2HB>I:->k#:❶OxSf3k#:؂آc(3H1/I`cT-I$`S(I,`SVhf((:k$"px +8[3WIga +H\l$.p`[Z$.p[H\Mpaq] +$qJYuN H=J\+q:JAudu5El%.;*g(qw1U eSx&MxF,.*A'qw-H+7xzm%. %$.PGxp`%.zu1eH\&c4JkOee"<.[xYI\$q8^aExqX%xج%.] ^A`}SF.+SwAKVJ'0 +@2F*qAbyI\:⒕q tœFSH:6aI:f xM BdM@(J\c d B>QQCAYP$ Xr Z⬕ Z؂AD*Dbz[1? ؂HQ0JH1xUR]H +KJK I\3lTKJ3At3Fda A M2+4rIZґ<3I\((%.4A u4!qƇm/q$HsDlq$.Hr$.H Xm%.69ߌd [I\uP&K\Nk I7-I\yf ;|)0o oF 2EMY$`'I`$qA$q 䛈-f >$xi7JLV&8cAkeXgA#0e +U( 4m8a}f71-}9z1wش??]~; +Xs.o*2Lۖ ~s7ҀM76K1|m/dX63h/RrA:_VqޖF>Ł`8_xƞ7.nHf~?a>]O~wbDA8iy3 *ZS^'9|SYί~𚺼c.momC 6&6,l 籭鷧Goqnzu{3R6%)7o}yPoS{m;qS6'̓Y +vx (A!A8Jhms'w|4og3x/+a^F@wIGq?_!W'n>8P'ŕWfVSKarn˻q"Mu׏6|k_c?kaǟk,2o?~?i\q܌o.뇹_˭}6vf7~kfSX0ݗ sdĕ:JK*Ggy?w}]cs?*P ޫFn=Vch}sD,~˖O:'M0jkڥq!gS~~C051 {7 μ.)ehklY~edYB6R[޶A稶*{T{cR9u{7őF{kmŹYuܑGxtlL|IE7 BǢϳgwaAx; <L2.u ª;$ 9 u$ &\h˸$dLK˙6-W5/d~yҲi9]Aܩωby\{l5\fmm}f&_Fb@pċO*yiRf`WNh̓|${h嚽wiY鐼5^'wbl]~,uN=de& 91;=QҐ* θ+db26gJ'<A,JKѿ3w*YZn:)z]8ĝƪGzӡu?VxpsC|X_Jt|Y0 wcJ"?9y! W;W[xh?c]/ӌx!sp{@ }+9}-vlC-:) +\۷>t(ۏN0Jn]R3OtT?vWOʨ;YP6rn9j +쨝SY2նÙ'>n[k:+;|L8/`dCX ?c}gϴ\Oy:Ĉ;C"=+d8{{ Z9:OփOlwx$'jYLE,(w;?l۲_`5rӤDt cFk + + +(8'vM{Ҳ݇"t-ļ QȢqI~n@tX29XW:kn"pl>0gavs`o<ĤC:>/6$/f_)\i׃A{M/9iʱ:$e,/5M|eҎL&SkwjosA|gb.s.&ba캈%+'ALO:p~[[d!YUv F%Wz~O׻VC`zŘ:VwI*4Sp& X7+Ӄoz讋-U{ \> _8 vAn-}|zmz&sQI}-8 E~kӄla0I*˵ ]rskTlR).5 p\ohvR'ioOm&vG\}(tCMtKӕ+L*<*|28~;K'X$.c/PIn[H yLR5zhA4js?vJU.. UZզ0Wn)wD/StGOqT8*aSۭ+^\ N-L,zIWe۾iw]mmP7$Tj~SC#X-?e'Ĥ~;j2YWO!߱) [#+Խ' "t4U PcTFZ-=a+x][elȩ"77EmƩտB)?O4 +endstream +endobj +734 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 733 0 R +/Resources 4 0 R +/Annots [ 735 0 R ] +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +735 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 627.273057 101.433071 608.523057 ] +/BS << +/W 0 +>> +/Dest (cb163-1) +>> +endobj +736 0 obj +<< +/Filter /FlateDecode +/Length 4668 +>> +stream +x]ێ}gyH8 lF0C~?j,QKwώfUE%DYKU"=_>~9IĿ`PJ J>,0Ati߾8{ZGHkοUֈCQ%29vQy%p)q*v +0æ8̰ZGFaA* T0C%Ci)Y 9VǤ Щ91tIx]t$s)k%-4oH4<&cD:7VyE%#eLҙ9N +V\s7 +{>03~Sv?~BY@:12iY7ui.msMW yiyUx\S١s۶Ğ8m]J~N|7Q^ +mm_G)`tfB/0! ^A +5Ŝ"\Rݴ):9S2.˜iAr=YhDn]btQ*(Ē8tKc}c@J]GEťzԭ`jYk2hhWphjP-4s84n?gu7:#jVnį4!6-_tg!v}4%} aVL[";L|KĐ?[Wp^Gg% Zo}{=U>˷pKtqq>g:WĬ\:G̛g6;@γ }۾?bkv9zͺƱ%xrvӥ?rL#ڦH;~k7|Y@iwGG;䑙+sK;,֝~l>3րs8nz7q/e +#}dtС߲1cDoEcuimHFWU߱]7vXɧ s4uSo߳8KXF^uw?uΆ͌K+5:CԆ^~ki<-]AgS>5-Ҋ!ڊv97%o9Gsa:b(~{Z7f<}fYFm-#s8d?F|/:GЇ6\5]U~Pѐ)6hez_lg/:|M16)VκcNM/yߓzէm qr^h٧k9`-rxC<fk_kKwX4̹6{?ǐѣmCĻXwdv9ֽ{F6g>gy~U;; U~co;786~.u+~aaz/jO^{轤qBٓԂm;7W*lZw9ZR}!ֽ{2s7}}'93dJqZHg}zzdg!uoMy/k#2:GCdx'Hn3y6w;{;v=oRxgm3 %O1Gs8no[!xGqgáq?] ={5QCroj;G2u>]";Yhȍ~߳vq^_W}?e\:CmOF . H3g'9}gnOw$RO#. ;;^$Vأwص\V_g-z˷kbgw9b7ٍ=F^ tМvW.ѭUx(=w[WF:~ܳ#ݑgYp{{>K%SoX}*$:Cc9ɯ+ >֌ @}61dz:C~gU *zB(sފ-oq4TyU0"8 SfxE68Fy9P#<_`kM!%C?|U6C~dp=#9+Tuy[м JCppX†ycT'wcĞ\+0_N~xWW'}Jڞy?jH~(UeбPsx /f`__u}m092 ǓJVϤp}Q&~9巸ZFl;Y8~\VUv 9~Lƕڕ]߮ce)d\f\^ +5p0㑯8_lҲ턭XIn.&ֶأj&ӮSdKYp /'4kz +,*ebTRWJuYd[̮XdkT1ˉ-7Ց +TlUSfJG4W{9"M8<ee$3lMrr)WHWGlxiXfw@T43A +vsD徲z@ʒR!TRQ23ҵ&ReDR1/T%XUUq?+P#\ +嗉TKc^z&;_8W|z^" 4Zծ5F0B7}eVv}&^N?eN>x;fE=_^jv]ǵl+Nt%o$; EX jKhUle]AC]ǵ&|m[uوTAURk}HĺBW+Czƭ/]/VGѱi ДC3&Xz98Mgڒ|&Q.ږth&q@OC`jM̀Ԯ[:&EMٶSp\`C艭=wO}4ٗګJei%<^jC?=k92y[s2LurhPil6jlhY(*]@&1XtpőP}&R;Qeūޒpd+Y}<]4E6oN.\mVJ̝k.*?C&DP,V9ew'zFzd)CXcdPۂ /Pm9UZ%L*ќe0crHÐCEWiTlj]NR( +endstream +endobj +737 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 736 0 R +/Resources 4 0 R +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +738 0 obj +<< +/Filter /FlateDecode +/Length 3543 +>> +stream +x͜ߋ$篘geIU`Cpa &[8!~-{3]RS%MϺϗev&"/ON֤`o)X?7Lp%p(J&S!dIc> `w=bf.t DcAw]AS4he\.QcAw]b\ˆ+l\EcAwHcAwϬ7^cAw!Rk:pb5pyCXw0 7,\WcK  ,yfD%Gr ccj,.{X]4Dp2U[({kòU4Uc=H]0Úܵ6ܕ &d^cAlJJQcA,:|.\B,U[8$֦P -Ƃa RLqȾl-5d&f)D0.zp2t.i,H BXX x#c,E&{d$ BƂa Ag2X$Zӱ}fX9I5`` R:Lh,f;h,MlsTl r r-zMo&d| Fd8tj,(.cYc֥7ƂBMlu[(~3i,( [RU[4e݂IoN._XcAQYc 7ѫGYI=.wC44j,lxCB~+8zb`ހتX*g=zh3Q=5@XYi"$曇qO?Z~KW~`k?}8n2y{;V,fkK۵^-QLmoeޡ߸! >L]6b}n6>}9zN~;ߒ·;}U~n 7\i\}ߖk2jmIKilcOڷY0Η~uWx}@&羚Y9+L73<0\;~b +K7Ͻ u&ؼ`kZYŗ8ն؜,myxŃ_ys̷y;+̡qO#?qU~N:ª« J}mVƵqOO`}uޜ'NpiEL͙8;famkb)POCnJPU铎uw}:ys>};e\cQ\|$m +5RߚC3J5ʼCNF>Uc+?wpgv~ngNI-|eM'' /em ̇+[; +jW?Ōa;t/8MՒ)xz ÝuޜWyw#j^yQ̒k{{p y%mnB8tԵ~+G:oΣn>vnLٙcށ##.7ښ=4ޱYC<1Uہgs3 `:o+?e,?Se;dKP۵C_JykEފ̷xHc<ȴޗC죺MO.M*}wq8-~OD:tn^Io +hyB'yS3׏n3b|0Ѥ|Ń2;%Sz7od[/mKe[XaUA%uu(SNȽ~i'׭֟ԗ#w퇭0CpO<':my9TQĮN[֝\(ھ;cqZyjbpLǓ+ldsvg_/$s`Eq)ֶՂMv{)sI +noKkB}Cj5ym-DkNJ5;qX+ mWb e{bGvujg .lCe}rB4KT<^m`TiPK 6OrR_4c2~5yoHʕZN]Njf*RW-u))Wdm$_~ֈv/2umSR:tVP]ongKSCYe"4gy㸹<دz'6x?H<66S/; K8^؁)ttAJ5O uM,X9 +m5uqhw;C[pYtnΧQڌ;yl| +/K`Y,k']mvA ci|fLӯ%y +M$',w/|}}o6Z֝xd|XWm B^6TohqܫZ5 uad eŖ޹{zsj.e:!<4+)C:쇊a;iC/8$} C!ut-w1A +L0ӲG1wfr+@ M4JۍOr-Y.#ڢaQKr.FW +S]5P4\@fMqVe㬎Rj9E:tg bg%J5lZFwNl/rɬXI*;k=ݐ3S۫CW5kg3:hrrP]"O>9F0kp4:J'"d a&'B7 bh Z-l~ਐkr .l$YlS#.Pf 8 +endstream +endobj +739 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 738 0 R +/Resources 4 0 R +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +740 0 obj +<< +/Filter /FlateDecode +/Length 3198 +>> +stream +x[Mda䒪Cp"0`Bbܞ ,sT|y?)5ԞS- Wz/)oU$j#QdGs5J bj*IEjHoA I95XjGQFDP#\Q$+"$ɑ +^|jƇjù; Z*Mt zuT]8,{Ón/|+|FY͈!|/||n}^L@KSwOlTзU4ϯN_O?0|">#>= >/9oy~3[pdޛ|py7*bX^"Hmf|>m߾zWv<ct(6}򽞶ά:OwcK[Yzu\֖{k{l,kdhˮIU(L}KPKğ)GzⱝZovݲ?WNͺFnzͦҝGȎMƻGC|e^۸[oQen|K3Ǽz̨)Rͳomry~)ATo?ӎd#fl1s:)岨q>#Iu2lvg99VO?ʇ&C#K,Wf~Ѥ^o$D_[ttxL]7D1eZ4+ootf6MmI*ˍ{ Cل}*ô?]_{Pi!&z.g}؈%ia'kgf-e}zvk +{?a%Poh[_Oz RXd- ܿlK\v9El&@׵hq_SxLNj(:v@7k^?梁U vޠh c#W9/4GӷQ¹=r}>f9MeIOin zzniK4S9?f=:65o{pGkek'Jg%Z*gj6j|_>2@dcx\.[7ʬ)-ef%x9ۈEƟ`036z@B${L/bl2.#wi&k~V9N!soi)s߰b60Dڽy^5RL46EE/A)pfܦ:6M=U{JƎT:t +rg;;afEv:y' $qQnd{w9ʍ61<ba |S,9o7Z~S<--H]}}=S^ֲó'OW<ߥA>ϣo>kޯ)xY$P66]zaH 2dNiU i$De5ݎ+ZT={m{# Hdr +#;ܧ>}?-<,4/XyZ*m*;u9؆1{8=qzMߓܘI_>ԥ}#kUF b9 (kfp;(+TlbEʔr?uUZS%LFPVz+{0$uZPT3gκdVieŦ3u6];~G<TˇBtY Zs.,w-z-Be; +2gDO/ⷙ ]\ZcɟDf|CsyqCZmY{(Vq@n7y] (l93(p#f< +xLYOݺ #u}"=s:r(4g&qq_۶xRX ?;;wC.{2[zgW#o)H#[~ޏ,ET/ v16CR={W|W^~-hz|9l}j<&k\SZ䙙ܖ%n-vjS}ڏcm+e<k:[ UW]*hށ6ojztք; K(i\)QUf;Rv~D/J*E:^Ga,@.tvMS-k*-LiP6 u[~EZ#ʖs1hRZ7f%]UOl:k\\^/]UUK Yd6?RRfCSߎZ%UMGm.|]? &WfFڽɾxK+WK/'JxC??cm-T˲4hzvزM \_)٬2PTLY-14K>(R1EoiJmD_rˈW%[ 9S-e% }*ٺn]Fkw -2U-[|w&T/^t7=8%g]4ϳV6&t@֪]Dz /PcEWTy-)ka)j?P_vP= > +endobj +744 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 593.554475 101.433071 574.804475 ] +/BS << +/W 0 +>> +/Dest (cb166-1) +>> +endobj +745 0 obj +<< +/Filter /FlateDecode +/Length 4458 +>> +stream +x][+9~ϯ36_$"B0=Bs<ݶIOe+&Lʗr粻g7SJ9w|8}<)zbuQY+'sH6N1{_%Oz/NZg3'SL1*/2TJ[AAP=9K*3n0%p$A`uQۭ* T0CgVCD QY#负$SHdQH:kZ)sg%-^;$lq6EI` V:bR.dA` ЕDlQAiIsrNeTi3O>Y$֓6ÈB'( l gQ[UFYt ocJh54ZybLt%0$ +$01N.'Ivr%LʓJ4`\Q4tFKD`l.HL`5lq$. l ,֩%.[tpaqm +N kX.{uF ڵ. ,x:$.p]kSXdu8*J\FF ɤ$.p]wM 1h%.p]ik (Gxp`%.| ,wu'iդSxN>Q'qa˓!K\ L'̨F3jK\8L&]Q'SwAKF“J`Q'IK\2P x{ wVUa-6 @a+i2` +IU=,J,qAҴ)qA$0I8! +6c dB>Q&\{ DlqI\z6' - -$.@D0e[\E\8؂HQ0ٔsDW%]0SkQQ݄XK\Ԕ0%.O.I\3Nₘa dK: [dK\H:%.H  UfP`ZtN:3I\\(T+M0- ?`&9X$.Hr3I\l1H\lz3J\lA'qAVa0Y8tJI\uz8 0.i$.Va%0n ތd[.=^=KּՆijdS{X|2ѶkRvfOfQG؛U2]G<VwҺfVsZ2P΀RޗB[joȝ6;qLH҅u}o5B Ncy׃[h?GSE#G[D6#"kGEd<ݢF(n<"oĄ;8^[Y.;Qo}~_~>6y F9 {m[Lbπc1 |=[avkE~0áFzs>5e\k>i0Iww=ph{+b$xe?-\5H䦷pȽ~Z# 4mP?~J}u`Ә;Zy4ckIGGPt=iw#P FDّ=ڗyD f#bIbyێ4p¼4"G?o"=ӣ<;V/ZƉo:G}}Z}C9kʼn1湶yamԦ/BA<=Nj>=yͨewĸ<Ǹw:G:rv5 >1;W9ݳ=+~k4Hˏd}}փy }Fs乷T]AftauKr{;H1<OsI"s0Nosxg3qzY|z<5Z'QƕNl9?9V߲O܈6-#F*C}y +y8x`quz!dKTZ ¯SsYd4`n=o,ϣ!_k`'`5[Gb{̽{Mn;hc`Cص%g5Zpg WO u#T㺗u :Ԕߝ:TFc#k'g{ܓבnR`Hx^jTv|QAf\leBګk3pe~m< zAry8$_qzȞqa-kRn9VbTц]+iyWL#b:ܱ{0ϣa:{~ׯaQouG\&1Ĕcy^wVR4ǙQhg$L4?㡂_)=& ?<{~W~K +MomF^cmc`_a߸w, v tz_Ȏ1&o)nڽ1.ZOX:Os.KK2m9a3tᶜ⹧6qcGJ{vKOWtϛy +A9PvAg}^WsVw(y̜wDjw{ae/F uI(myeqTvֶ$qqI쟫[]%ϒWw>~9%z4k[N +ҦśInۣ̂Y'gz\=cm:#,s9IB.sxP\-p=?JQ [DU#SKXQ:tsTު]bexGQ-L֕dt}:(s﫰 &4~6{MB}ĦAh</[ }'ơK,VpuOgV$(iҡue/-[I4דw #v,/J+X_=wOPGI7+,2/ۜctQoP+˲g[ӇF-cIS϶$RWeJD2UeXm +K$z)K=y4'z+aʭ3^l $N- +endstream +endobj +746 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 745 0 R +/Resources 4 0 R +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +747 0 obj +<< +/Filter /FlateDecode +/Length 4211 +>> +stream +x\Y~_1L(^a`@yƒ`@C~b7YIj*:*r۝-~>u3Ԝ|y{zw&G?梱ogg)j* !\p?K'o7&̢`rFU.(M>&kX#ftdeAD1 +]8*ّƂƬ[ % +^h4w&5^&erijRJh<,Z"djgCՓCTC9 |RN`!K!Ap*JY@ aAEÂ%[xK% bZ 7D@.\jXةaA"ܳbn4,@jX@SA8@G\ p785, .kXaҰb|)VbIA  #enஏkX@]O>iXG1V]@Hհ<4,`ƣ,Ѱ w]ESbhe\IÂuSyո +q -Rհ w , 2װ w]@x]հ wQ7D ơа8 UyZ X [$*/8Ӱ`l,P pjX6aAZB٣ K(^5!lFjRh#wș%"+‚Pݏ=l4[~K$9A +}ߞlQ&A~2ݑ`=Z4$ؖ{Ö`it1Z&A~8al;DN\ɃEϩv4^NSAzm΁.l\zvnkk-M$ (+.eCYd>t n|h>$騚ŇCEPA=9vL'҇C:={y_%rT?e7kx M9|O +,V,{{bAJ@>s)%'6&Posyq5mHRlI}W>}nDlGAߗL;ͼak}HOe-51̫ +XGMaW/NPN +W2D}ķhd>>x{ ϭuZG 6zFk5~£/y~91'O:, ~yux(Ƽ{;oVQ2@qSاRt)b}S9Ϻ;=.s&嫅ޥ>׶>c~iyb_qǮK[mkknmRM^Ӽ>3j?5ۮ6$ /UCgc_$'diVWui]bڬ Oaȴ/[sIO{vȐ}#zYir%Bn9-™!]n!uTqn[f}_m:Vmno4tJsX]է6qM{o4c:˜ù ۹'Nt]柎/qD. +[ygq^ y?v]#d8?O!rW rA9|ha9w;ˢmhjF h\Xgeͮ#oQ}d<$y#8ǘ)G9/DZ{8ĥAm=зX8-qJba>-ֹ{0r|Seǐ8V̵;d/+84C#7vi +B?"KM]Ϙl(9Qg76^KtӋvI'^ߺ7RيaX+NGF +5c&T]Zm-jƥ q[r&w_~oLM;w_b[t5[tiGFmSΑsO%E*߮{B^ b"Gpo{gH!/ur>!΄e^!z皯;tw[Q{2;~pzKʮG\>PQn$^oxUj~.f1"}[BH^8 S~ܵeI[RT[DwTq;|?GP]QW7ۭ^_l? t̂_ـL~vGH sz|Cwz}}~#^6&wpT=1ž+mooΛMh“?Ogn$ʑ<]؟ć7>5"wYi3ОBMV<5&xQ~k0$(OEFZmC?\wB_=gg=tO,ޟzv‘t y6'=bz(1uǀ$gG.XAGM.$ѥ0oE0mST*յ!T[аBJOm`VgbD]夶[GRy*k횟LWe(^]w>_&bĆqKm~)҈In7[lj6vד bLj.]UN'*2KjN±9[%IzCV<|Vg6T2Aϵ 5cKv!g`$.8\aCrюɒǸv>Hrwj!W|'^Zb=]I&ۑd8(LчF:Љ_7? 2㩥9烬iHVY_:~ǽhn2{ʝd~O]'DO'ٕ6voy{4 콥Wԡ xxiq2j>~)B +r9,fJ>!qMgJ 9(.ńuJRN2T4Eą(#2L4+ +%pr`rrҺGz0u@>HXS.rN_{N 0ͅQA- I6D14iceJkj)bR*CQZ%=j)(JwsuvdQhIT-C=۞2h +fejgxh^>>ӣ?$ĕ2$jr9kKJWe*h}U$r&P6;~S.qYV)[u'\{Ug;lB2V-Bl> +endobj +749 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 265.954475 101.433071 247.204475 ] +/BS << +/W 0 +>> +/Dest (cb169-1) +>> +endobj +750 0 obj +<< +/Filter /FlateDecode +/Length 3991 +>> +stream +x\K$@AFc 3kX/*U3;d0;]Ue/-lM%E>>Ȥ@W[l.*p[bBɤ\LJ]m{tl8NXM)QPX,^a`Rr!*h XƂ!sV\rX ++C sV.YXÖ%aʃ ³D(=j; ׀ciBwb5p( "pWܦ1 e A4,MI)jX OEaA" KANakS|$Ys$Y2hX[M E R԰Hfraְ | Zxљ*/g)+ 2^pIe R,8*k< YdkXH:/εg{>WENʱCECGO2 3Ry͇UKXuo +uo"BnQPp5~͗2/rG<+e OOqUGIׂx78U廎~i};v7o|sU$\?o;@zGr:ʎu?}2`s<׿I3x`׿q聈'B[[ n-~O ߍyvF[in7xcũoZv&zѶhIF{M#|'Zs<4.T J*u\ō9=,BUNj?EqVi;^Up"Uk;';dwUHbQd|Y;[:O^Ӥozݟw]w?US,-sjc^ԭ߯?1XJ~ׯdϋ|UηkOXN7su]7_+_iG(xl)?gr hS;y:dY\e4}V*-=cȕv<[F-"ϞyqV:.=&W,~134.,u^Kw?<\1y؍}>w6b6Ĉ@ZKǶ}lyLGǒ/t{y>䇼/cSl̻-XHoCd.D3- Y0]7-r\k)Le,_oblI/ٲ<SPLrEӻ+)b pkҫ.Z?yJ^ةg58[‹3)IqyEq+:Xo'N(ᚯDN v:'xC{|bz,a}\\Tsˑ{}8L} k2 ݭ[w9fVdpm>2bc$:y1(dl6٪X{|g}Çj}A3bmn{x605Urvb0=+d$rF׏bqNɔ}K8Z?p|tFMrlv%6L3s]Blz(!6%Pv 8:^%[O1in7yNں4m%iR16T.uIҩ}64c3;5yl!fHەN!juw*Vyt-MCa>w+ae:#-Y})OvΈC~: Lo&iN4ePSnl4yG_"t O`܎>$r6;y;갦ieS6;fR ]cꤲ.D:vؒmNkX9 9tS~ؐ@}h's}A;fr;n.owFbj2ŵ[SK]qVzL1?Ϛt%yWY[y-ٜX= +-ν;GKhsvm2xGdJ[8k,{YZS+!6&|}!,HIȓ\WDZd a{JV^FΘjQԻR4> +endobj +752 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 606.123057 101.433071 587.373057 ] +/BS << +/W 0 +>> +/Dest (cb172-1) +>> +endobj +753 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 296.981993 101.433071 278.231993 ] +/BS << +/W 0 +>> +/Dest (cb174-1) +>> +endobj +754 0 obj +<< +/Filter /FlateDecode +/Length 4871 +>> +stream +x]K$9ׯ3?/ H4iEZ8WfFUV?Hg9k|eȨr t|UJEa џ!LpQm >6s_~q׺9[sƟYTL1j/2X8彊 Qb bH1('/ YVq{V6XdѐXED + OQ JH‚1hUR\H +[JXK@ IX3dR[J3Nt3F‚d! A M2;4bIZ83IX((v%,2A y0!a懃m/asDdq$,Hr$,H Xm%,mr%,㓰 M0doZ$aAv&Bw<$Jxwv)yY3w1 !<"v?%Y5;;^և뼻XbQw\XNp/Z_cj6:]&>u}E^ݞ匽:]).)뱙KLҢS.[t1HuX 0Ɩ4Iw2tD7W=KfZq|((⺣:{cPf7)GۭnJi辡xA[g21`Y(>̊f[u6uz&c\.Z~Үzy,FճU~Mƣ]Ե449waVjks0]_q9;P>aO)&=0+lHƓ|2x%T]ƕ 5AW|Ds\Ner&NY޿?0q:N^lÌ*Omʧna;d)E21[aH4|t  QEc-JʁbNRK/?-tKgf*I䯌6|o5ka((~D܁e8|z;Q$dDGu_/<4?8e Af>CMJ1O<'?SrD<̙M:_0:M8ZUp:i>ØjWI3(ΠHRU'{-[e]qϻu*X^;A3|L:J {}[{vpրKY*itG&T0jpNp%H=d[{/ڵd=YzB :uny^k}ls&'2n~kjr?_ksU^S"C V+}nLɾ귎H%FCQp u9@o?etÜu~͡kIo^[)jt?%쁢ӪAvC? :?b4!cˆ_Gd?ᙻuN.e`p35ڐGygC.eoYi uo>:ػP|2cmnTjƕC}B_Sdg}^h_ҹܳD4²Kv"OۖѫC<٠g u 1d&Mwδ/?#j=5ɳz6)/e _oS(oB0m8:rw'K[Hμ#& שM)5+5K$_a_`,A<8o'~`f䤇 *?|l4~.P;|1MW^VLuHߴ0;*ڎ/~p~_A[Q'Mx RϬ/RS2!&J&:{u +BHiJ<\vdi'rNG,15wZS +0YOytIP*;_TU +䩠 +endstream +endobj +755 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 754 0 R +/Resources 4 0 R +/Annots [ 756 0 R 757 0 R 758 0 R 759 0 R 760 0 R 761 0 R 762 0 R ] +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +756 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 713.673057 101.433071 694.923057 ] +/BS << +/W 0 +>> +/Dest (cb176-1) +>> +endobj +757 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 373.781993 101.433071 355.031993 ] +/BS << +/W 0 +>> +/Dest (cb179-1) +>> +endobj +758 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 354.834971 99.933071 336.084971 ] +/BS << +/W 0 +>> +/Dest (cb179-2) +>> +endobj +759 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 335.887950 99.933071 317.137950 ] +/BS << +/W 0 +>> +/Dest (cb179-3) +>> +endobj +760 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 279.046885 99.933071 260.296885 ] +/BS << +/W 0 +>> +/Dest (cb179-4) +>> +endobj +761 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 260.099864 99.933071 241.349864 ] +/BS << +/W 0 +>> +/Dest (cb179-5) +>> +endobj +762 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 199.152842 101.433071 180.402842 ] +/BS << +/W 0 +>> +/Dest (cb180-1) +>> +endobj +763 0 obj +<< +/Filter /FlateDecode +/Length 4696 +>> +stream +x\K%qޟ_qD 0@&0!U9|n>sFx.*>WEvu3+2[sIϗO?Ȥ@WKln*Ml ? +'Jp)Oow;?7ֱggJΆބޤU"V&%Š`h,;>Ƃ\VXbxNYbdYcA [BRX uD*w +51WT^&71 Vc Y1d %Xfx}N/B=?{> ec`_r^BPgY""$|c7 + Ejp7c`1[>>_ I;${m)MjB #]Suf~4YeEpČEW^yw>^<}8!wӤ'o Y~<}_N. +Al@o`׿q聈RDօneD( O?/RhǓLSMr^K7q4[f~7wq>ӮʯB\ByGSn(p'{LNi mR*%ڈd=0U>4%KbD/`=nѹyRecH Q٤',Kհ>}}\E8 hҿ]u}ia믶mp-3^mmޏߗ@~{zߍ[cpN먍aYCm}IzbwPo1"8_Z/!tcLw5y8cl'DT=~nm6uY~0y{̜Ōڤzy5lkأIJmYeTϭEId4мTf+=PU򶽫Ӑ +vctO7~7ymm) 3tpM%B/b Cp(aA󃁸(иL %>I<_iK`rث tqf`x]u~2;zSi$ \m1%y*٧{/!}վw{R["w5.+mλ>ӮB;a#&Y,p0ȇ4 `/jPSiRlCO0wrH=(CA;^j8dSvh8 +5m('MIC9܄N3t b]IG_yhs269|%jhaޕ;r4^== ˍQv:E\^!I)X$ʞ6x=V^Qr4윾{~Y"B0q϶b3D6گ'?hsyrD<,%Ӊ_wM:)oʒHòroj)[k1ØPM:龹1*J?<[ϱtk>lߞ}XY^VR~d+j˯ǽ2xpр KY*"е>cIBx+i:\m| \iRmW,/Yǖt?A ݿY3 K'`%7lazHIA1r>> =#.r~>F&(|F s2D# i;{4sk1Ju\RSfpf `X7g#C撳~b;mScrh86%f}o7+k5Ҝ=?>!q:bnVd%YXM:wMFNbݲHZ%p \1u}n^‹)jdDL},Im Jos4Z ՛ZO{x^f#/πsLVZi#GϳGs4m+=d:?%k<e;4{Qϐ ^w:SQҔϲ>q65U5yicۿd^kX{31֧mo'>^Cm'ơZ&\s.c5ikfUF?ڷY<\fT#[햴k׭g&NDOJ÷m#PϯvoKtTyk^^_{祭6o-5}ldol9d͇؎zvGǔu>߶_ǴF\:WR+ӯm1uƴf=q!ms&?DL 햽vy[f =v~G6"g=Ƕo="y)率,ك2 ] +!X__̕_?#[ΑH_:] M׶|>$/Mq9ͧsv. ;߆w9m^ʚ`MLl{Y2Y1S/Vq &3f2,r<6M*֞Ԭ,A"^qYT'Ǔ<OQ%gn$b<<KmavJ/郡rxm?^Em4R?j*0S+˰K +5=edJv!@Xb%&3AˎGŻdi"#qg vgi[YEЭSU|,5մ$TJ >O +rR_Qk1?͚|ra!fH˕VN!]Njg*RyT-MCa>w+&Fȷ-Y>ה'q$類a`fz3Ovra/Ϊ*,F*B + ݤnt"kJ鱣k={픺7ŏt{鮱 uRDO' cK:Y cL&M}CrT΢=v9v|ɩϸ/ty{7:6ޓsduo7:C0 }Z;:WXdǽrnhV>5GY({?/0,~QG ױuC=|>1?H)6b_6l(5t?܄<Vv;Dvtġpxi2;7a/ xvG +7pp|ĝ.\vSE̍v 6bAΝtՋ0vөm9>owh姵{3-xs3DͣZ ̝rm:mhhdk:j)bP2k:̳ҕYnx=Z= +91:%{,Zkw*ю}m0xGdJZ8| ^Y;NR> +endobj +765 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 653.523057 101.433071 634.773057 ] +/BS << +/W 0 +>> +/Dest (cb182-1) +>> +endobj +766 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 634.576036 99.933071 615.826036 ] +/BS << +/W 0 +>> +/Dest (cb182-2) +>> +endobj +767 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 615.629014 99.933071 596.879014 ] +/BS << +/W 0 +>> +/Dest (cb182-3) +>> +endobj +768 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 558.787950 99.933071 540.037950 ] +/BS << +/W 0 +>> +/Dest (cb182-4) +>> +endobj +769 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 539.840928 99.933071 521.090928 ] +/BS << +/W 0 +>> +/Dest (cb182-5) +>> +endobj +770 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 478.893907 101.433071 460.143907 ] +/BS << +/W 0 +>> +/Dest (cb183-1) +>> +endobj +771 0 obj +<< +/Filter /FlateDecode +/Length 2686 +>> +stream +x]dϯk咪!CpE` Mccz \9GzafG=RU̻RjNv~>nr|5 +Ų/@5. kϯKpMerY"QU+Qr1UC`FYa +fX +P^Uf썩`11@t>IOLKJu:өGtեba:-g[7-%3]-T2l Jt`w-3]Uk1 +\uAEU{7QX.$Rl*K!wg[BFulAq1 cE+Tg9+@Yq@C< +l Rlj` *0–`Xa0(pkR=c&>1( 3@A\- V +*X.=3(1vJb,PpW chIutt2^2cpXUu؇{ŅRP GeTwC (`!1`8Gh4!e,y]P%iRcAw}Q]u>cƉbO2Dp2Dp,`É "'5Ӂ>Ta,.X<o .U'ƂKP]vH0v8 8+cAw%2$pW a-q tp[V]XawEre,Hn;6 RnP `C R1WsN +t.9#cA"EuŲ1 vUka,h9 \iz[]h~`"cHJ5c,1!ì2] +d +t>3lGg,@. +w-\b,yc,U p3- B=cA  cA Yf%1LgXP,*2MiT[ OK#c$Hsh4ƂRBK7 cA)%H`,lff,[1TIɌ8t" -,U&R0_Gھ[1ػs񨷷cޘӺiijݙ^E9޶Nlw_ν^:q]m%˟0w˘ož5bqsG9\&Htq,׼NUr~O^'\lw8dq!0m[0LN<;6s9tmֽc_G6sWeCD~\꽿w;--=߷bR>cӲ}=%FjݙN:K 7Z4|GXdb)~<—6-;. +e#.q~7(]ÃyXٿu\cDf؛1j襶珎-ы]g) S!/|lbP8XǷlL]a:u~Y^7e9?%nV7*mW|U}vu.F!XDD׍jbj-Q:o$jGD׍;JQ̦4enu-clSۧ{qj+Q7IxۓLj +}B];E~RT58<2ެ4;i9Ӡge|cϮ勐 ;2e؉{~c.h  I.5wi{tA-`x=ڇ3C.n-؟b_fw8CXxmL~*sq[7x]f&~Bx}޵Q>8t-V2u:JZyCkaiR̶!N7m] +m}-%hE:.GTAka}zg1wirФCz[c`M:ǥ}q9zisƻt=AtZ]aݣfgvj]vi?(. .oݦt6ӲY#י몌I몤ђ(9ɬ9bBdim=6kj=m} æ!iF2ŧ_Ŀ/íտ_M#jY<&%%0r5|Fߕ=a9J 7(+Wͪ! xʖ_BhsKۿ}i zK: ݯڏ]2| +endstream +endobj +772 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 771 0 R +/Resources 4 0 R +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +773 0 obj +<< +/Filter /FlateDecode +/Length 4231 +>> +stream +x\K$@ AaV>̨Ȗ0]Ǘ9U΄+RjN||8}<ђ#_kho%J<BKt~<ΎKg_jM%) +`b\r1,V1bfGV\KX/@م#B [,jpAb๸$(=Yb  쓅pCf6D;,X ߃烅lHK,,]W1 Z\‚u3yuq7Z^\J‚F,,]g`GD ة-pJD.haaqH,,8뚼P-‚,y /T$&//p$gax!$.jaAR ‚%Fc K(5!lFRX<.9Xwr we۴ e!ba\{ R,,HCg҂haA&" Ala륆Z,,-, eRo<,p>`ȂL R*9&cЪK~)pm +x.[X3/؊HdaϱB؊⑙[<Yd[XL<,,( *3ءQKBIbaAHha&A?(6Q<"[XPrXXP +d-,oJ-,YYXP)-&[X‚ +M .,@7Qq[Cz3[XPD(& iFPo,^,5%7H3Po"7ԛ<. +dx]d\Dfk,RqĀ{yd@^V<3 HDeYyjmLG'}D{fɲȊ`dv6uzY;)?Q0 k +k8A!(&E~ 3"to pMvjш"#:8=U]it5CV)EDO(P^cmt渓%^R޳?ߧ|/ >3^r>S%)'%p?ULK+ⓗ孪|Mە~~c{DR@Pr"WX,Ϙ'иyylLwf,+-3mI"m߭t][c9.%E/!ڿclAMSnim/s&krj,^*XL!wziY\dlNI FfsF1m~}'uu7uܒ#䠹X‹LD`5nzYt/t߸I[4ei1ևot+;UDۺ_a?*qg N]͟\ z~|^BIq7-]gjƆMB>Kænt'>s3XʷO8zv#.!.lx}=&|oMoo|6qx/sVz|rM}Ń<WXOiYw\BH>ދg>r7:d3o?8$?)ɊWg";.q_S)uV?yњx3t{wr\&P'촂uc<ң:5)Gk;{:'lOAs-;{::{뿮FMn!ΝzOE5_k?G^p>tSoZoHe|MŃ_#m|覞򵞞;GjB۵{zW؏<7q1[z,!=ן UN~!y(*5}6tgMi_Kj~Ÿl!Et=H=4,lg u|hpxe_cHyaP(g-,ؙ~Z!m#l>8rmJn9XR1VWyD\ͷQ->:جub]×fxJP@, Iy.,ϣ +캙\S;?ѳvh `&u e /~)I~T׮A J:a'$oSaWVWbģ] dFRy*k횟LWe(ޠ]ԩNWпr`0lME?/#'ݳ{l'VP쮧A;ȅLj.]UNgn* +/KsjN"9{%izÖV<|V2 Cc;R'u,3R &9W~ܑqj1YrwGMN^0*K,_$ivs2E&yc#h$/v_gY;S|^GgYo8fu4lݑN&o+wp8F=+؟dWfؽa+ٝ&n \MSzf J#z0Mtp2yang5$e:BK5aԓ ՋMd.u sCwZ[j8 =0;=icvDN=:}P5u,Lljhi9ڃwjhT셑И.*jݶQOʝo.@N#.SںuSKyw騭2Uҡw-EyfˑKtA2tqnw۳ZOALiv¿Ď8ժWkWQ&]*)CX^y[?L0!PvNd[G0 5T_ݭ)טRvB;YJEynȫ2=0 JQY +endstream +endobj +774 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 773 0 R +/Resources 4 0 R +/Annots [ 775 0 R 776 0 R 777 0 R ] +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +775 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 488.554475 101.433071 469.804475 ] +/BS << +/W 0 +>> +/Dest (cb185-1) +>> +endobj +776 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 427.607453 101.433071 408.857453 ] +/BS << +/W 0 +>> +/Dest (cb186-1) +>> +endobj +777 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 115.110432 101.433071 96.360432 ] +/BS << +/W 0 +>> +/Dest (cb188-1) +>> +endobj +778 0 obj +<< +/Filter /FlateDecode +/Length 3943 +>> +stream +x\[, ~_1WJI !B0y8{:!?Ӛڜ$TJJt;[~'39|ɚ,~jEeW +gg9J(TO|Ÿ˵Oޔ ~',&rJ6(uB0)VtK,EcA 4`89: +ɱƂ[ e&\0J;sE%\.8WL1dǐ%4^,1[YrxP$ ω4^,) +/CXWVyF4^!JfVy &d"3I*MIym(*^daj` z G `(ch  z8Yϓx%+8ZK԰ w]b\hX8HÂ|f 'װ wx]հ w7 `!аq%g p5,)lld`HNC5,]EÂܵHÂܵGA^5mFjRh ɂY*pb XB%kXuG>$#Ѓ,By.>1GTGՇx^GG_7_귿?<G[8",_>In:sx˷NPN.6ט*p%\.oҦ$,bx p#͆]kǥ0 *e쫷Uӗ_~T|2ᗟ/DYQ^FFO_xsb !!jbK6H e]{+>yܯU/~o3G 5e.cw&|rT,2 E"H/7s-QخClfTW~N.\s5i&R,:k1H>dcc{bV&v2l&-txuVh.Œrêܯک²ʼ~MG?o_rU&)XG8 +!tDZm]AR8s-;}\&xFz^6;r1f#}k.Lݮ-*] &6_lVm]v96\[0ηaG/s}8ˤ}tC$YznSeߐnu=;]ݦk޴n#vS7O.gU[|=`ÞB-ܱt[r}W_堯@aJ:Snzdǹ9fVvuNwnG?6a鞞䚭/g M<<p:v7Xa?Ikۅ.}07ag^u]]v`=' yyU}t׶:v-:eݯznw~T׮FPePCo96>)I!~nЌeA!hWz;t9haTZ()+˒T'vG,{cm^Sቓ| +Ii&VMo%2 g;eRSkv.7*S8&%̫9 @48Ͷ\JfW}l؎l'au,3r%.0LyGrɢ[=q'9|^i츓犆c;R6&>${;N9B&ys%wo$ߕN'Nozy +^>15rel縷-y=KdNn=/?.϶[8ǽ_gw;tA0u0]hLC\iq2?vo恽]R_!yKu9̓GH>/CJ#\KG\rLvbA Ջn\F%Mcv*&iXfNZv(Bٽ~7ch:&x:e_3'RyX g -xwBͳ[kIvbhƈ[j)bRj:2]2n?FD# +9:;[X2+T-ӎs~~ڞe$nMQZZ qy>Ɉ]"O˩lRyucEJ֙lr5c`P+nŪKzn>?Tg~\ B2I(_+Ef6dǙj'k Y +endstream +endobj +779 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 778 0 R +/Resources 4 0 R +/Annots [ 780 0 R 781 0 R 782 0 R ] +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +780 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 770.673057 101.433071 751.923057 ] +/BS << +/W 0 +>> +/Dest (cb189-1) +>> +endobj +781 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 458.176036 101.433071 439.426036 ] +/BS << +/W 0 +>> +/Dest (cb191-1) +>> +endobj +782 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 397.229014 101.433071 378.479014 ] +/BS << +/W 0 +>> +/Dest (cb192-1) +>> +endobj +783 0 obj +<< +/Filter /FlateDecode +/Length 4574 +>> +stream +x\[,ɍ~_Qφ Eq30 g1uNYz }ؿOqLUVvOϸaOӧ+(2+UR/?]^_i(779$ .r˹寿c\9X,r*X,m^h}X(,1g1|$K{VXM+A GѸ=+C  I,aha"h"g*@AYB6V9B$*#6Coă,Rxȃ,6E'"dNx ȃ,:h#S>'")=j|rNKив2!d 6%, 6IX`+aAHrA‚5xD^PD dX@.,BCDP%71|Z`2e]4O<`\%}mwt(`p +#r;ԓ}]gzϻˊ +SrB>jP:6# uu4i:ݝtίp}]gzϻ8yi^ Sy EWNlw\Ab?"#^$+ he9 h1HVHNܤnAtlӏ}0Cm&7f{W!Tԙ(x +l#jBHorM_ЩNDhk4b(@|b\ e?4mgͣ/Rt. +`iG!o+0\ o<ЌNy7բȩ_m?*xwb?fCL 'Ocֹ|k9h~^c}y>l"nlz% O,-K0Uf-9\P h%ͤegE FA_-ן믛l +Nĉֶm.lͺ_mnnIwrbHeъMdoQ˱^԰fAĿ}8*}u=ۺzd ovY}[fcO5 f2cmgxMaxrkW4gIv6uaԼF6N%yh7,Q eGrtO=Rv] ?4!.p[N"g]b*{iYzNSxϕ.Kw lqۼ]G^>nȅn7^vX7.G4ծuFOȰȂxHWd{'2#.( u 1?__uy[>1myaޜhcXKx=G1Ƀ_1iiW4{e}:Sm2£"ߡ-S_iҷ΢X% Awcʫ i|1V m"W첆vR7-áv}FtGLӦ0yk_6J>ҰNxx16h~x7z4LkmyHo~X8v(Rq^.vʄRX=];P|F›`W*'7U@@_Ǐ0-u}>~p*p~&_V:~)=0}>Zq;}mR67* 5\bݵJqcmn_q+p;|yĉ{?Xg3GcjĐkl8wgy!6+Õ6)zM_ߊ&(MR9o'4,yw;Yyst&2­A6n#?;KBGpgyhf,?({ly|4+]߷54ml5;af5۹~&9ڙuuC\Μ-a[O(Č>ժ=My{Σ;V8Ōg3}> @ǵhu9[q69Pܖp\G~뗅o^)p5S'AG~/G7g՚Uŵ޸㠧u}䁙5מ鿚3Z7Vt-FX1=Pm>k^b[5>Qc?uѶ^֚=uSO;i3~?ɳz#*AYn~dS^PCRbvd6Xsӳު +LA{¦?^L&/6BN] ˅@&G> 3~},2qF]-W~drYo7GߺiIF|tX g0{2^+Sꫤ\ +] in;zn-5y *. b}Vq#iݖ"R;m];MBNe$nnC6], z ElWG}boF4?DRW@M-G~#ڵk2? +Znxx8/~`f䤇 jLMU$A*|2I՘}J顣I0GI]&nLB{޺klET6cn1HuXMNfkX9u "9}wUdL:vw[8}V6ڒSdݘl" +<`vt7 ɤvsyOgRse6\Gl[W$Wd~%a#v&dF[߬NxMZsg:ȶ5nҌ? 6Zݛq`/-8~E +rpύ#: +n:tpNq1i<$[9OUӼ.0/8fbjdIJA7*rh2S˜Or֗qaCC},7ApZbщFϓR&mE tc)nM-eIR1Š}ִw+ϻvmyvhhc$ܻSits:- aDzto¿gSuZR 1i5+r)`"4>ǵ}EJ晰Ke^ȏPr%NB)hU[YK]eT_8MȨUUjJ(_X +endstream +endobj +784 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 783 0 R +/Resources 4 0 R +/Annots [ 785 0 R 786 0 R 787 0 R 788 0 R ] +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +785 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 797.673057 101.433071 778.923057 ] +/BS << +/W 0 +>> +/Dest (cb194-1) +>> +endobj +786 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 736.726036 101.433071 717.976036 ] +/BS << +/W 0 +>> +/Dest (cb195-1) +>> +endobj +787 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 409.229014 101.433071 390.479014 ] +/BS << +/W 0 +>> +/Dest (cb197-1) +>> +endobj +788 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 348.281993 101.433071 329.531993 ] +/BS << +/W 0 +>> +/Dest (cb198-1) +>> +endobj +789 0 obj +<< +/Filter /FlateDecode +/Length 4800 +>> +stream +x][$q~_ςSK lk1~m0s G~Yyʊ]m/ȪW4St|墖wՊt] q?%/q.fI).!JP)msKyA[LIR ^`$ c0RvJ0M +f䂠 +vh!At/^|L&FY@iIｋ`KJK:[&RAybU ;IEtEQ[WZ%D]N:⢵Ji8`ӊ:( \uYe$-Ip6(VgUVK:[ $` _ba +^̠%&!L E  +J4`[ȧ U:%" +8X [3WHk` +H\`-l1d%.`[Z$.[RIK\`paqm$qJI εN ,H=H\`y#qZ^ ޵޵5El%.;.VG+qwVAb 0 p%.V&*&` uax8܂rwSF2^e(1BX3V4%8NX9 ZtL^'Ң<.[xy' x wAQpH\bt]mw78 E#ۢ+ҢR8.hH\UxQL" i @ƋB*qbyOxrJ\:bq tœF(t]1’THxn^6%.! L.>ҒBC'k'qAP8P-HK)J\ lH\ lA q"%iDbz[1? ؂HQ0ycft-"t:H\-XȥI/q~0:M[0U$.RD\i؂ [Di-qA4ѠDd^I\$\-H<3J\/9T&RЄl; D#`u$qA&QaQF`3A₤` "> K%.CIG䛆$.f Y|)0o9 $-!-0oI7NH}73o"b7]u $L@Vg23; +밶[ +܋*{ +s$qf"^5 +Vⶁ"'O|kf &!9(f[˯߰^_ ҄ +$ Cq#64XTmH, "l,,xuac+LƢ62!sv )`d&֧<49 &HA˶~$[le>CҌ =su@A*+T9m9ɘ#)PvV>hstL}w4X-&el2ys#=x~룄~Gة~w׿?~g ˿"u7?h\wQL\XDt۶ʛ܇a[_jrwB;qٌwL_~q7vق$h+!6"ܝ-[-CCLu4JV[YwZ#p`oTRnIn-} !m{+U CLj;qM$ aa-=>2_m MͶ~AܷK Mq' u}[YfD  +OJ.;$=WJII0!NZҔx{ݫg~z5 KyˌTnB&le3e6Negݻfcrȅ*7G]8E64v8uęˉ=pT3mAOtsVY3GCӗ:eae6_R #$rH|>C3?6 !m;}%џM;eNl*m pć&)׃!\CqL9qy cK7mz?[Sm{>Y'c7oD&CS+u޼ "}y9T?/" 7ӻ'c}bodWC=؛b&\mW9k5fع8o |J¾rͷ6޳C{ɾǡ>l˧exK> m Vl;6 On`]e{v],=m]ݎ7;}k?f^5j۵OQ1,c[pv.ös>?a}H0Uw4jo+S[XoG6j}l֣C>bϛzNjM_q5k04̝9|UyjmXy䧠g?9g󞧫{uw)mZ~SU_CۍZׇ̟y)ˊ_yv3nbgK0z-(elT{i=]sVدo"tuDcK8 G"wؚKKbjql~Z|,VȖa֎!jzW|m\l *&t{b fGu!si6 l8 C3[Sͻ2RVqD A lc06 Q/Z9AtjkT⨚)9K/fsDq+λ}Ud1;! v+kB^4`dJK3:ljtfo徳IFY.B,J(IH˘GY_(; ]N#[q:[έ1_Q8-ބ}0!FC(:n k VRw]˅nMіyo{ c3ҶC76c׬\ơcI&q%Z2".xda HIw}P`aDCGcM7Y V!aҼӴG0ǽ N]Ʋ,o/cK~&Po˲g{3-hs1ʤqTli[(:]@MVkn)wE/S _DGvۊף% ݩQǒQy^KiM}n!~DjTs̝δ j?e'Dq~'j[KP>U;Pd[#O9+c!kU?בM"=V#WKJ l S+͹!/CNbsAi$f 7 +) +endstream +endobj +790 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 789 0 R +/Resources 4 0 R +/Annots [ 791 0 R 792 0 R ] +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +791 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 228.873057 101.433071 210.123057 ] +/BS << +/W 0 +>> +/Dest (cb201-1) +>> +endobj +792 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 167.926036 101.433071 149.176036 ] +/BS << +/W 0 +>> +/Dest (cb202-1) +>> +endobj +793 0 obj +<< +/Filter /FlateDecode +/Length 4069 +>> +stream +x\K$mׯ E/`!`d`` X0[#5#㕙yit%HFfZ|_fgr.)嗋5)X-vM,'\9\%J)^_.p67qWoJޅ~I9%V:RX!| ++:% FHJK bYaeAYbXcA WBRXBUy2Wx.G睉x.yGdbb+&cu/@ƃ,L9iY(UyD/A! +)ñ$ :Ӏ,d 5$ Y ְXaw=a  w!AO V{ Fd YaAqg Rof +Yԛ(-L,)ʃ,H$ԛ@'!/aAQͬa ԛ#,fR 4pi԰7jLgf8zb`U=2^DXHDezj +걁"&lOU9$̜dnG1ѝfrG_78?qZj-9v~LQa"DB>"D*SHXln"$B1mn/2$2͋Ra(uꈑ!%B^^e tF-ky?*k_,~q`ͼ\6=D$ϗ?}ç"[[LQHKOOh|ik.CH?[Y}_q6GXmV5ĩU5/m6~{ªgm.e$0m]zeM2JLJaqU= k)ݞufՖiӏb/\emշx㿦#ζbo&49Ye$.صxkwU*-7^V^CQ5XpuS͸tokE?݇[*V W-);a)qcʖZl氯. ɀMՑ@ rވjIu;=b9+";1p=q|h@OQK7:xwb/WlA^xR}^hn{wܷu:lG#&k"]~f8ln-*=-Q[U<6QaҺw^w&)?sNP8۷/-fBk'y<6jOz<ÔkGuQ?9{ՏF۲*==#=;r>ݮ儉Bl${x}/wtZzF` Fܳ]Pk'v&.R)GN-ycŞ-׾ghiߠ|2vMK[ݝ8ii短m5Nt[ckuA+_^p"~v~sK +L<%_ .ncv}M>▉W".* v4Z*z˱!bܓ%!{mb7D]OKHJ0†nz+q9i/rW\1x0> +endobj +795 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 582.273057 101.433071 563.523057 ] +/BS << +/W 0 +>> +/Dest (cb204-1) +>> +endobj +796 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 521.326036 101.433071 502.576036 ] +/BS << +/W 0 +>> +/Dest (cb205-1) +>> +endobj +797 0 obj +<< +/Filter /FlateDecode +/Length 4092 +>> +stream +x\[c~ׯг I !WKиrneb,.iXϬa0z p<:_!hXs0oY+Ɩ5,]װ^ $QyccjXpEÂܵHÂܵGA^5*gHa5pT4,]q$5,Qװ f6%aA,<|*/\ !Td 6Jְ yBkX Fd YaAqg p.%7)P|3iXP(* h*ʹ&!:iY|a Jof f DDI-wC2i<."Өa.o"r{ ̒qfb`U-X`/URk+9Z4,sTp`: ,0\}ZKT3sE Ft?rٌ4;~[8/:<|V+ecMqb 2u.+I\v^Uȧ?)ɿ/\l޻wyu޻wyCHlKyơϫu^eE /S<}jjȾys@v(7x_poO<~Zڐ;,]5Q!j4᳴x1^|AڏvR }?OEy-Gڄ]sg!:Sku:|{Ewe]Uע*6\2w;[Eez,yy8 2{jnuNߺE^ɼ߮f+f[\Iu ?{up~+S[k1\Z?Fq^_~o3umfW ?Û +4S$j% "q9s6"+>ϋCC\Ȣ)TdHl0}ֳjƥ0Dh 'mEo2QYW;1&\)-R_P>vshc6w_?lC2`OI۷E7[֞1vŋkw EqXh݋jI^ωY2sڦCG!yT8~ ۸qCSNKNNO''n5Ӫm]6bhG]ikAOpZ+}hvXu5;Eȸ>݊8|,hb:N$Cُɀb] 9p~aZ6Se;"R xL4z{&oVfq`.aIh/]Fyo$g8~3ͽK}m演g"Qߛ1]_O˺1Iz~=3rI6Lr..mtn_?G=+dKfr~nx4YyݣzM;^;}-v˥(T}Ty?Rq W\WZ|[z;a*S5_PC6 U [5q_5(Ʉ|ńHnH.YXK e]YmU1G{?SYmc=bzS_{_IWV>*ȹXEzgO08chw}vRe:7D,B ݕ;|2}}߻c&S=+IoUV&nr8zomϧVY-6߲߭cfWlͷ+rK{^נ>vuw?&QQORGNٳ IWzKZfG飏̚{V^qJ@jۭ)`7,$D);&O ~/]euRi|6p s +RZTKcZ4zJp6eݕ|D7oװ۶X_p6 <~l$Oe<@6Xwҳ2ѪXّ/ ,rE짯>k+9~1ɏIX!3#Yci'deK86>Now(z7moD +=ۓeָF㳡v(!vAv8;>Ivdb2${ȝJc{./uWչe{[.iR>T׮FPePCo96>)A!~nЌe31wCѮv +tP۵ة4ZٵtOQSWhŵ&Xdm5bi7~G<xIi&VMo%2 g;eRS4y,D?CETA0)yk^IlVI+=aMl˥4lv՗Vi/;Zmz*]<[V7a]?+WB`IÔ7)gG51Yt1$ǁ+9wސw|'cK~L 0F+ցN#to>_R_!yBKu9̓G%!q%MK렸\&,Sb$C"?[A2rrIͮCwZ]!0IFE&r'@q uNzP;54Ƣrјo.V]yvk :)wf(\f;mLuj+Ϊ҈)&UaݕİJܵq(l9bɬ{PyN۹=.)H®%gG}Jkj%d05Dꏽ#0:4>k%H3a; gt(O3O݊U<\}ZdnS-mׂиL*ϴY~]&Y^9(ԿbN +endstream +endobj +798 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 797 0 R +/Resources 4 0 R +/Annots [ 799 0 R 800 0 R 801 0 R 802 0 R ] +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +799 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 698.673057 101.433071 679.923057 ] +/BS << +/W 0 +>> +/Dest (cb207-1) +>> +endobj +800 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 637.726036 101.433071 618.976036 ] +/BS << +/W 0 +>> +/Dest (cb208-1) +>> +endobj +801 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 295.229014 101.433071 276.479014 ] +/BS << +/W 0 +>> +/Dest (cb210-1) +>> +endobj +802 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 234.281993 101.433071 215.531993 ] +/BS << +/W 0 +>> +/Dest (cb211-1) +>> +endobj +803 0 obj +<< +/Filter /FlateDecode +/Length 4914 +>> +stream +x][$~_QφI`w eه}lx*Rt/[Os*#t I>E(3UEK)x~rQKp +#jEaq:?%$g]rLvӯ.[Wv׿e%T *rn 8/^Ĕ$p mUfPNR 2& +3,m3L$`N.*`6J:t-%ы$QH:kH;-wQlqIiI`DJ1H:[JtNt]Q啖t..E"QS1ĭ(htZ/ش3 +䃠3+DlqVIga ៤O8bfmՒc:؂xآ(3ha^8P2$vHQbB!XLr $JCgDVG^ba a:bI\` lA -lK@tK*i ,X.TX0$.\@)I\`)qb ,X:o$.]KK\`"-`h%. *H\@]L##qFubbTxDEx,.,V&H+7x8e$. !%.P.(lJ\:c%.@a["qO`E%.p]1|.-:Džy^^IBe%.p]g`E58X%xW$.p] Nap$.h芺 Z2`^THZJ\Xޓx"=h$._0.ʀ)$xWi*˦>D `/H\#-)/qO +:|/ve:B$.c$k #X-p-%.-1I\G<^t,1E]N BK1%./.I\3$qAHSE,EK4%-p%. D[ J\MDXK%)3®&4&$.p$.@`a-֑171&ǛA₤` <> ơSJ₤#MC`^r%.HV!D'd=ǛADEuNS-8 &IybI\x3J\7&[TpX3cF&L wb$ϗ_`/;e-y x'0%G0l^B +LN] fҍ).߇44c u(j[CW!;]Ulmb04iA.>= +%UOE!}凜[f } pw#Ʊ?-iĸ0}g7x~ϊ/uo Gh@`4E e[Jhx8Psѿ9m:'']E1`hy7s<i?^nSG͘= 3[1di^IH!)6ސAR?;{TkwM5{#LlTv8~K~f0ln/*Tak';1 d#\'A\7;##< |cpÅ'Ƃ iIŠ +Kvcm)x'< :Jk,. G5d0%:fY9 [t?1ռ3~u$PG1zyL'< Q&E|}Y>ڸCo𵯹ĊvH3yf.Uf>̆VFZu= RZKRmclꐃf4̑{FBϣBYE\!}۶ 59yR 3 ׼_Yhxu8JnqF?$qΓyhڪr&DLz%Ϟ@IFGØxyn+8 rx_QcDtߪ_|>r[;ͺVc0-)PywyǾhºWG|<3QђG4L\PFt<;c9D5p?ިm\|a4E>Oy:iysm=۳Iؙ=iG}wWt~_Tգ6rqhA)o3s}mgKޖz4ܘ?ŖQ4 >{gQv.ӡGtcv١Lړ :8\gCh6q b9?]=lc9f'86pjO$=㧞Cr1*wP$m!)TZttQ0A}+:V)ʛηhSĦUޤ-ߥMy<85ʯoiNHj aQj:<U{ר&QI]M8 EjфlaŻ0V QKj:AtJiThbϚjwS:ml/ri_戲Ż#VqoGm&v!\\(CMUKr~Ư/QSkRfES3ٯz%<7xI7!4v.B_:*!=nKzl%lTcj놺&f +Ӹrjbנw;]NʼZ]wF\V*免f1C?Jبa7 ŝF߲N;kdҊ{咗9c |e ]P\F50[Wq%L' +Uئ 44~Ӻ35{M 6 Fۡg܁6bsdul}qi*\e:ۗP$8i_ҡ%3dX(. +Sle&ru=OPXk]G}܊! \`;^+M+{?Eei7JD7ߤr׼% 79֩Q1wROil[7뮫uhuC|LU6)?;⁡R/O)BM:XcO=8akyJ;"ud ñ*h 1ل7(cPTy19[RbVP7d^cjCE(No_J>5o#V[(- +endstream +endobj +804 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 803 0 R +/Resources 4 0 R +/Annots [ 805 0 R 806 0 R ] +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +805 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 638.673057 101.433071 619.923057 ] +/BS << +/W 0 +>> +/Dest (cb213-1) +>> +endobj +806 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 577.726036 101.433071 558.976036 ] +/BS << +/W 0 +>> +/Dest (cb214-1) +>> +endobj +807 0 obj +<< +/Filter /FlateDecode +/Length 4504 +>> +stream +x\[$ˍ~_Qφ/~0b㇞g)2.LUvtÙ+($')2;2%"]o?\~%梲PZ-1LiI%/_.}-Dg_E~I9%VXBXRr!*h,1B"GGVTrX ++C O#@ ,i,aKH +a1*" Kңsvx69`5^Yb!k<,d h'!ި; ׀mi$q@ +5x]cRѰ w9ljX\ְ.Ȃ1RRbCʧaA2 @ʃ,H$K%kXdiXdAʠa2XdjyY.j}@@%" RǠUݒZelҰ %ZMa~tI% R,* -!3x ְ.xjXdA6aAvUf4 B53©&4*&4,}A2GAHÂYÂ!3N&כIÂb 2> 3FÂb3MG`_r5,(ޠD#כIÂB% EQyISQ-0 &IE +iXPbz3kX4&2zH sn(@&Ef5,MdZmo!!?ә+ά@o@lU *g5zh1QY樞 izl` Urr.Бw*I>$QR̷}>\r޷}>JV?Yi5'd{&y. wm|q?| mPڤXOǥEhtӧ(N E@n{8 mPFqs9\;}^5s9@q?~4˹̠">s@9nl:p:|'/x~:ImJQd !ԯ=~pRN7߃2R2qC]tc>ߚ6>j PWp3,LƦq#w^jg06Oϴx}oovn5uti{)LyƘ1?ls\"0zCP:7陔Z-y!vݫVvvk׾izRo_emkkkm?F;6-3WE$X(cWŘ[o6/MWyj_wiϓx+^u6N|Nu>[a~gFvU>vV몣'&w*%gjr멾A_BOU6^3f'eC;{ngǵ/zVJ=V鹵-IDncQ7̠t( E{QWkn(t瓡c;{<^O]ch f?ymR~nc5td3ϊ :`[xM_}[}9?Bm:i7Y枉=eŁvl7r*6 +[Ӽ^_Ǽaz;NyIUO3WzˊZկD;z[b&O@xWŚKVnM -n➞پ5a[e[y{Dw<-܄lF\qq)dGp]] :Jy(^+й{kHd+|ln%||Oo/j]v4-yL/"}qMNHܓ݅qd{MO {sIjJ{_t+nxnn#nyټPkWʛ7]7\7[MÍ͛ߖ\Msp|67_MwOջu?oX,%x1bg0C$'Np`} vV`iw3am}qTeo`\Ư;*^b>qF5j[58!yjI>~\z;ezSSSSoC7lsCNasoFLqq7 +g_9co怟0 6e>֋C~~787kZz5!.-f2N=W-ߵRu4i66yK目vVD%񣮪b8 +֌'\حxw༱c{78T ͡p=V5cT׮3PePCo9 BN +vQߙ>4//oX;vSHڮhg5?E_UOۑh(wM~%؉r Ii&V]Ow2)wUakv?P~4\__1$7y$IuHӀ4 +uٔJ}&Q3CcPc%u3RMMr!O9تɢƸvHr.!W|'j`=]?I&ٙd8(LїJuS}}L_.59g[~̲hrX3- YF2~c~#_؉c#ll78k+aPp]gL8!vZ{m3z0緀tp!*yvk :m3i.@36FL]غugUKi 2U-#ȖHt4WgfˑKfeAwqnݣU, qffv __w>SU'!6Ϸ!PMBkc0QG#9?+99̱&Y~5. u\Vp5RGw֧MиL\*='ߠտyJ?F 5 +endstream +endobj +808 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 807 0 R +/Resources 4 0 R +/Annots [ 809 0 R 810 0 R ] +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +809 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 426.873057 101.433071 408.123057 ] +/BS << +/W 0 +>> +/Dest (cb217-1) +>> +endobj +810 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 130.126036 101.433071 111.376036 ] +/BS << +/W 0 +>> +/Dest (cb218-1) +>> +endobj +811 0 obj +<< +/Filter /FlateDecode +/Length 2862 +>> +stream +xM +l@e 2p Axmry;V f5MVWȧȚ *Z[zoJ"[.vxta\ R,ϳ˵Wۿ%rt'SRKTR +ĔUGlST4꽪 m5y*1VGUahaW5 RqTL#!*q +=]kDO'rQNZ9FlIJ2l,`KE<]-]-` ebO$VU]]jH)*B9$:D)GaK\lIB lQ<8Dkt X,Ga !Kb)A[TJЪգ@-h)a 4q@@$ a"r]l $4X [m@[ U@\I ĕcp@\;=p/ ԕvEkX (wEp-p`c@]a* EEX? +!J ܍c` pu% c1܍wrpI~:[XcAwasb +\[X]X>W̸97 F%  |V`|=$p;w96X] c!0\յ@V *HJ#8P<dpǂ "quEjX:HVEӁXaO mz,z,YpXVJXA䱠 ruEz,@b걠D"1z,( eX 4F +[)z,Bm Jʨ0.P p*t\<[1{,@-*c,]X3cXPlAx,VjDf XL:f5d j23y,ZS锬ά ` *M`cNJM9HRF+7ǂZaKifX X(K=y,h\QoFXzz,hB7Q9q$[Y<4- )-H`Mez,h9[Y= @=VoM dt.27z= ?*ЛGb*V#0eU(cnOU;왵C6"5l&hv;|/?g{j?)/f+HlXn*) +L3}-fBiD#7316vl05\:P]K?R"aFv׵/Z'r7RػQذV£~[9-~-tjK$+E;"i9v<.YRn#]m_7߾]WzC_(r;kC!޻Mmzl~xM)Ӷ]h{?o9()o1k:\n{N3DF;EjV=t_6[c;酓}=%Xoq}z{eqWLf~2|,r9א:plCPZi䅹;/9gGFXz(,es,IlR-n7Yn77Ws{ɳ>@# ojG%$O|-L?c/}RϗR&B:8r b jLO,_>p-b۷~6Fi`on;]J#=?m%:bf&<>K7o["4FS͒c_v3؞b"90S١mg,ӽ8jcL]a:u,o˲^7 +ϥOԎ{W}vQ^t,F!MX䈮#|EkbjvqH:ڵ]7[+UF:om})եwdb/1B?V5*7NxP0iKi&+(u[;ĕL) +c]f%" g:\ʎabsϓz3Zfe|q%Aw3v.SK?np姓xڙF&e^=~8v>2VLm;]| Ʀ2őn7[0@a'FvqA'@8q/۾!GVeahͲ_mzIHq$^w2AAD G,¨ l7}<ƅ:o ޒAyhWaspȲLlG"[fnطKaNUXudnɗK\YӹCqk\lӲ c~]BlbI#RN6veS,MzvN8Fqxu?L=L:>Oto>uȽNXsи,V*O*Œj)~ dewh +endstream +endobj +812 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 811 0 R +/Resources 4 0 R +/Annots [ 813 0 R ] +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +813 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 797.673057 101.433071 778.923057 ] +/BS << +/W 0 +>> +/Dest (cb219-1) +>> +endobj +814 0 obj +<< +/Filter /FlateDecode +/Length 3998 +>> +stream +x\݋$ ￢ ,$ y,!aw0C.۪*uO,yg$-OßBΔRs-~ꤱOGg)j* !\xͩZaw=aZ w!Q/ V{|аDÂuOiq& "pLʫeX\[XٸD4,]gѰoQ2:_!jXw0oYʫR4,]װY$QyccQjXHUÂܵ1 p]Ba4Uc=Bw k<வjX65,H hX `aA*djIÂT-x(T^28d Bdy1Vv _||Ra1bigkߝ}%煀| %~j!*>/X?tֺÇx[ж>iʋ "b Fw?]ЃMzpE )n"ܔqϛy5 F_C-_x-yp,˚3!+EhFk w]6_$>_{Zx>qa ڴvX{8{[[Y#EV# )0(ه5Чo'(-~W^qX<ğ}\эoI|JsOK"NZu~a+o<{4#|m|fFtϗ^?pN[0Lu=g԰mJ[}[jɖMuf>o]`fk#p-֖^ݺ0.ck=5t-<ɹHxNv> =>_ڦo},]c; on"n:4]y<V`q{_}]Q,K5M5Ȭhz+sDk9\Ix_^_2Y~,o,]Y~K 5T~#3 Yo+/>mQmv#ng6TrGe*=yҽF9l_ 4PV2Jed +nú FS_1܇}mcCEGћ:d.lQY:RڰlyOnlQ삜V߶>D^[iU>·#Pg'FQkÛW7kxo羈/6\oW|փWmvV91V7ĎXԅ^f ]­~ht9?U [x—,.V8bLi󹫸Q7pcNOzrIgƾ^]mg}v/pu]ݓdlnVg,6[;p=3nA +|=qCp3^nq/#:G{M0y_aPGcr[DzGHć55;:d!"#! E/v+ٜtX+r'i{ qd]M7\Ugy}wҨhP|&E52w>Nt9eh{fdx_/Bm a7m_=6>a](+D\ܞ̣-d~@1a9KC IO]D+dO] ӥIK91]g1׈ [Jǝ{SݺZJA]rLyRH|Ǝ 9b3tG!WB:T8f*2/S +3IEσ$sa:7~G<"Hv^:N$ ;eRQ(.Bhv?rP8.ŏͳ $iip0y+Vgb+0c&u3RmWMr"O=U8{,9q;i`$1㎟vw%6ڒKdnv&S1$O}l.pqO_-}{:GeO@ǿ{ }p&Ӈ]+L^)(Δ.yq!Nd8֩JMvH0=Or*%NDahYtQAȆNJEgr(c`LSC;Mӑs‘OzN 0ՉQA-E NVD14iuJe,mER5Ť*- #F9+i'70 +(9)9jɢВzPiJNM{=aT$NkbGsZm'ĕ55|h/ҳ%%`P25r5_̯=T! +`ԧDj!gNv| +Eo7d{wZQj ut +endstream +endobj +815 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 814 0 R +/Resources 4 0 R +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +816 0 obj +<< +/Filter /FlateDecode +/Length 5429 +>> +stream +x]K$qׯ0ۀ !=n FdfFUjzGexg{3 #|;[L\ROO$X=?'YJ]? !$ +'yyٛRw|$Q0)dYI*H.(I&%QIKђ 'OJJf- bYI#PraT O$ +'%YἵjD+iMDG-;sEMKHski!ȱҊ1rpNKcŽjiBNZZ,!ASdhejd:-%ii6O afP0I1)iDEM,d!ҀRh2 NKc}"5 552K O e +\ 4 Œ4!; ,LQC ai%Ұ xȂ Y< Ж--4,[.jXjX!GҰ rJE N @=iXװ vaArVN5 @5,:rа٤aw1U=@WjX@]-iX@]`Դd0W,7p׳pד@Q Ǫpº;Y# PpK4,`+0ʘZK԰.Դb\B+l\EP   ,װ.Rki]հXqp4,}3BuմblYf iX06~ ^4,]!hXi%daa*cΪHwCki]kSѰ wljX|ְx^ÂɔXSӢ3kX, R %Ұ)d ,{ ,p4,G`Y)jX$3iX8"CjiћjZFKDiXXd z F% iX +dPѰd\KsNt朆CxdeFa,4bI4FV̬aAhU԰&RM, ?l~l"Q K`Ұ g/fְ gװ)fҰXOÂb7Y8Ҭհxӓof J7)㶄(fҰda(jdTq LB t"d4,(1J5,x:%%Ltp IK3&<[-^N,gVg ,a[) 셵J꜁JN@LD90HtV6_cÄS?E 63'yɂaA[̱d W]ӯ_`|LAE n pejqTJ!RYtiM)-[ Efg"RpuY Y/'en3$O_ǿ7ғ-$^ 0OpIs͟_~孄K@^%p_TuN' +Yf?.M, duε,5Ԧ2{Ay ]v s6h$KKW7`?5%_4w+$ekoTSQbZ ޾f>Msn.pץ[ n F^/iZ¿"5ͬЈZ&tX)(-˧@gt"792ݭԕ ^j,ZTU=]F b]b-v.|ER z%;TY?K<c(.n'HAS'xb_;Tp˓zՌFx_?[k_T M` +ǍCe2++c])|Gߏ "Qj0B3>WfN&E88$Rqq"^٣X5-_QhdE܂( 3cyT#BbM5,J"Jog,M%Wa %ApY9d1#Hkmp=et+ d\ #(U8>%,4ˏ!dKI#eF1(yB JQ<fj?a ge& +:܈S%x[⫍A^ucP?wKXB?PJ{YDK(]+pxs4ܝU܀Ts6iMo.+oy#WvsZф,g +$k?,{83i܇uc6)C֔R. vjUpԇ4GS,K̅ܽAҲ=/̬[o]սnߞO{aObUb r^ؕ ц|)t͑<[x_.-nxCVŏNx{no6Q0O]J^=wUחGF[eI끈^)AQk. HH+Kh)g/ <̀d9 +K8kﯚZ`v`Yv:yf%/Uvn"r8\^Xz8 +Zn67l,9bK5mM̪G ] pvF;SlVh^$\>rc<V57n)Ym+Zyz-.{߮C{,"׭(,NF $@xG^AW){tWmඐ$W -y9tvy_*#@elQG0E_94]3r|γZ62h1HnW+|s2 +7ku|!yF)x2Qޔ\nC}y: A&G\z:u8K|N]7g5_axo skv3G"RgEJ#ma}ݺرyҜ 82{i[Ib95f+hu}s_ESH̏*t kf0aJD={7P)2 {%̇ z$h zJ$HLZ{mzlQ96gm} T77|jsy@)S҈ P\Uss gFŶbcAfcSIX|ŗ!/5= S?K w3z\tu V."z^noItpV.K,1&hI20p \G+;>0캊{OE90Va^/v &r9RvepE =-dyWuO6)cu߶+6kup[:oWۼѤB︯^u!>Ox.X=~p;sI1|pO ʹdȗB"wgs8Zy!$/›,xSÇ:Wmv:?_a-YT?ϕ,d fiuE0B^`IR]fq_O;U3_K/^;?yOn\츷 -|d> +endobj +818 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 766.923057 101.433071 748.173057 ] +/BS << +/W 0 +>> +/Dest (cb221-1) +>> +endobj +819 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 747.976036 99.933071 729.226036 ] +/BS << +/W 0 +>> +/Dest (cb221-2) +>> +endobj +820 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 729.029014 99.933071 710.279014 ] +/BS << +/W 0 +>> +/Dest (cb221-3) +>> +endobj +821 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 710.081993 99.933071 691.331993 ] +/BS << +/W 0 +>> +/Dest (cb221-4) +>> +endobj +822 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 691.134971 99.933071 672.384971 ] +/BS << +/W 0 +>> +/Dest (cb221-5) +>> +endobj +823 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 672.187950 99.933071 653.437950 ] +/BS << +/W 0 +>> +/Dest (cb221-6) +>> +endobj +824 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 653.240928 99.933071 634.490928 ] +/BS << +/W 0 +>> +/Dest (cb221-7) +>> +endobj +825 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 634.293907 99.933071 615.543907 ] +/BS << +/W 0 +>> +/Dest (cb221-8) +>> +endobj +826 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 615.346885 99.933071 596.596885 ] +/BS << +/W 0 +>> +/Dest (cb221-9) +>> +endobj +827 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 596.399864 99.933071 577.649864 ] +/BS << +/W 0 +>> +/Dest (cb221-10) +>> +endobj +828 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 577.452842 99.933071 558.702842 ] +/BS << +/W 0 +>> +/Dest (cb221-11) +>> +endobj +829 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 482.717735 99.933071 463.967735 ] +/BS << +/W 0 +>> +/Dest (cb221-12) +>> +endobj +830 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 463.770714 99.933071 445.020714 ] +/BS << +/W 0 +>> +/Dest (cb221-13) +>> +endobj +831 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 444.823692 99.933071 426.073692 ] +/BS << +/W 0 +>> +/Dest (cb221-14) +>> +endobj +832 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 425.876671 99.933071 407.126671 ] +/BS << +/W 0 +>> +/Dest (cb221-15) +>> +endobj +833 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 406.929649 99.933071 388.179649 ] +/BS << +/W 0 +>> +/Dest (cb221-16) +>> +endobj +834 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 387.982628 99.933071 369.232628 ] +/BS << +/W 0 +>> +/Dest (cb221-17) +>> +endobj +835 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 350.088585 99.933071 331.338585 ] +/BS << +/W 0 +>> +/Dest (cb221-18) +>> +endobj +836 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 331.141563 99.933071 312.391563 ] +/BS << +/W 0 +>> +/Dest (cb221-19) +>> +endobj +837 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 198.512413 99.933071 179.762413 ] +/BS << +/W 0 +>> +/Dest (cb221-20) +>> +endobj +838 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 179.565391 99.933071 160.815391 ] +/BS << +/W 0 +>> +/Dest (cb221-21) +>> +endobj +839 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 160.618370 99.933071 141.868370 ] +/BS << +/W 0 +>> +/Dest (cb221-22) +>> +endobj +840 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 103.777305 99.933071 85.027305 ] +/BS << +/W 0 +>> +/Dest (cb221-23) +>> +endobj +841 0 obj +<< +/Filter /FlateDecode +/Length 3621 +>> +stream +x[[d ~_U% $ÆBv;6a6C~>t쮓aOIuQI:ǿ"ޕRsÁ\Ǯ_GO]5x|.r:>v:ÿc1ZS[e \rh*X`r1nV1b ׬ 1h P V,5BX,kA`|t JO/xJ e|٥,>zW]J)'%V/BEA-^,Ld %ː%@;*&P"o*XɫQ(Xy"lM< + %Q&D`~-p"r.֠EBYY(' ANq€PIb@(Il@I&^0A(BE&p 8@ta h+ p+T BL\XX\R 촰plau9`avYJ,,,.3F<@)[X ]Q`؊  ` `Enn,,n,,8pt`-,@/%DயO/5YX`L^u>CBΧT-,]* w`CD DgDj!ZXsp oLUG "p,,@vT$&/;`y ^-,H]$. Y ׀5|ɪx]a%‚հiaAJ RWsNJ!3yɡ-,Bd<"Y,,@b*baAC ȂdzdfyY`.f} @2W9&cЪK 6y<-,YB+pz N% x^,,U,,@(r% 3x ɶ.x[XPdA6haA Uf" QKBOHbaA4UnJHi `?(6Q^u%xQ ~][;Js# yg(pQ^4#",ps,jB/A; uɮh.r#67?^˾zwy߷?U9(y.$ۯ~(\ߙ!9)S*8+Y#d@$yw]&/"7zNXcC>ʂ~ é>#np" FN:wù R~"W0ؒǛؐ/f[P:H^/4QMbm#j['hY=D}lȗ=wSqs7ܬ˳˨UKjEj?ij ίΦA@>#](-2+/'NNO☷ڛ7v!Ĕ +yո mlx]Y0wK"o7/0)}|b&L'b>]7xnfZn8sQܦ~MϙRcc>l-P<߲Bz>HEl2~"RoCkDaQUbu4k+C/xF.j -{NWhYs4׃Y +*w_:OgAg{rf3gP < MsTTc5n}<67|O. 6>5^Qy\U'vINV㔚V^%/,ö8Kޘ䧃-Yd;d%KsNH=PLent|TNwya_FwQVeqN(k9xb&Ҳ$oe{}U_ y[%ɤjէ 4hU5cCdA: ԧl/=~3HٯNѫ̡k~2MPA;-I^/->B/d7^SY&\O_"?b6zljD^B^&5*f[# qPy /äͯٚo^I+= K^컨O+uܘRTޚi1sSCc.^|G3]ck6P:uQݢ=P :m̘L jZ#_VMuW+骯#Ȗx$@q5> +endobj +843 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 799.173057 99.933071 780.423057 ] +/BS << +/W 0 +>> +/Dest (cb221-24) +>> +endobj +844 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 780.226036 99.933071 761.476036 ] +/BS << +/W 0 +>> +/Dest (cb221-25) +>> +endobj +845 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 761.279014 99.933071 742.529014 ] +/BS << +/W 0 +>> +/Dest (cb221-26) +>> +endobj +846 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 742.331993 99.933071 723.581993 ] +/BS << +/W 0 +>> +/Dest (cb221-27) +>> +endobj +847 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 723.384971 99.933071 704.634971 ] +/BS << +/W 0 +>> +/Dest (cb221-28) +>> +endobj +848 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 704.437950 99.933071 685.687950 ] +/BS << +/W 0 +>> +/Dest (cb221-29) +>> +endobj +849 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 685.490928 99.933071 666.740928 ] +/BS << +/W 0 +>> +/Dest (cb221-30) +>> +endobj +850 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 666.543907 99.933071 647.793907 ] +/BS << +/W 0 +>> +/Dest (cb221-31) +>> +endobj +851 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 605.596885 101.433071 586.846885 ] +/BS << +/W 0 +>> +/Dest (cb222-1) +>> +endobj +852 0 obj +<< +/Filter /FlateDecode +/Length 4344 +>> +stream +x\K$ s+HÀ $`#rmflrGUQUoٙ"(R;[|'39|xt&梲r2(TOtŸ\NqgoJޅ? ɤ +e)LJ>DKX#$|d%%12 N1|r +!),g!֪<b+<Lң\|Qy <#21 Nc Y1d : KpVA4^,ɪ%ų>YCH_Xdxi@@, Uy +kX@D3iX@@[hX@[iX@[.jX@\i5, . T4, `.;5, .=iX@@] +kX@]5, .0p5UdkXt42iX]r6iX]װj`Q؊9[ e  |4,`rw}^zQX5B@Xa3a +?ŧV5,]1}*qoeb,.*Ұ wY`5,]GX5pb5,] Aq4,f骼blYÂ,y +$*/Ӱ`l,P "p{Ѱ wm Ұ w-#Qyװq xp[ŧkfXwMEÂܕeSÂೆ +̦5,łOEaA @ʃ,X֦P$Y{ ,4,@D`C Y)jX$30kXBD>jMNksIÂ`)v 2NaHaϱ@XL5,5,@ sdY jX}FXKBYÂHa&RAIH~l"QykXt3kX3dkX۔|3iXP,dAħaAHa`mrO9ĚkIzF2NH!G*7kBfB* +yܐ 3XO}ᳰa~#I~~o~Λ~iu-[i2ππr|'ᲫaI)W;J?iŸ)9OYD] epkvUݮ-RHY߭eV歎 }V_ikeI3eUsoUU6lǪ*:,{wx3j/Lŵ8 KkRh3X!pۉ$>xͱlwM䟓=Wb1hǢ-mڜyx|6~OCOpeqǿ_u&pUتɳWIVY[=*7cXZ4P^OFi謕ClzOr{}U]կZ{ݏ$C0PsAÇ#W2Z_ph?jk#cеjo$i'̪~Y8laubqs܏sy)SK7qMkخʦ_ 75&\dSΙK<ӭ r 5T&ؒ-ǭE5Q8#OKo͟Gw!khPܠAzv^9Bfl] l4x9bM(K's:‡[6io̰FuD|9[׃ݼ5}ۭ X%;39ƉGmV'2˚}VkrrQk*kxfoz}=lP7a{W!opP0 cwĨ5Nu=Oi/ۈƝ:rˀhnp}p~&WrxxoCBL8o#O3jfawv3Zn@vlU(uU437.2A')oa=Aݓ\;"'.`uӤX֦C$6"=MN <ؐ)kûi[K Јh= Ңc{p}͂Eb =eo&7Gz \5v \ >176\e'vmgm>yNى Ώ" +j[5=֟gs޸{uK (aW[Hm;[Ѽh-S܂vx͂<^X #a@U'!4f?/D߱ P qV3[FRwy0kLãҺoThmyٶzq99cǶ i {}To]{7> ]Qh'zȍU<dzENj w5!ܗ]F5봶*KUu!aZwt HL6g}>,Q+\ţޮ ݴ־'y;16ô}:XfHw7#y(gy8[bצU"k7^}֗`Yy/<]qX~JF䧓"x,mkj]]IXx%[F#C&bIJl~X!ftGOz(!64ho粣Q&ّHSw%]uq4Ot^#^N'*iM5"m4*Z>!I! ۚfLfO&/ {)I-L^*i~2LrWBLiM>b}ݒ%smd7@wOz\X: LNvˠTl4yY.P:U~qMJ^Y0F'⛝ULuXӀ4Nr)uU ]cꤲ.uo:vؒ]NkX9m r!9Cw`{gMz;v>g'R6ޓsdegOl" +<`vtw굯ɴrT9OˍHN^4GYo=~~9eغ!+|/I5bwyAvy :<쭤:w&ÉzGfeع rImprAФ<\$&q%Eǒ.".Fq9uXd1 NE~b +#2;7huS4_ ;-yv=ȱyP;wLj> +endobj +854 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 432.754475 101.433071 414.004475 ] +/BS << +/W 0 +>> +/Dest (cb225-1) +>> +endobj +855 0 obj +<< +/Filter /FlateDecode +/Length 4378 +>> +stream +x\[, ~_1UU I!L0y3>;unLxm/9˞RI_Ige&"/?~TKr˱N^ggJΆOޤU"V&%Š`h,;޳Ƃ\VXbxNYbdYcA [BRX uD*w +51WT^&71 Vc Y1d %Xfx}N/B=DazQlIz*5,:oװRҰp#avr& 2 u GK>5,` Xvrw] awa Ha{װDÂܵǟ@+cs>WMив1 pװ wY`4,]qRkJ4,( YBY4oI7N/BWXÂY7ѫ%,fRPL4jX 䛈lܞ!!>әY2άV p%Jj̀#E2GjcLKjـx si,QEC™,-}9z3J>?N_<=?ϦJY0~Q . +pex*&!~߈."B*]D P\Jn""b>صW$eu8-Q.o.)< y=Ry]Qet^nVr05m^=l}6c}\(//e XwX[XHߔL$ / đ.wvy='$h勓 [U?p\.ĴKCSq4 ::şŴO}Iğ~ߐ:.\?27b*8鐧?B~)}ᨤ5?[A]N$ oudnuR&b:O[He.-~',EtĈlp3RȩY@ciĮB[I a`O]w}m ۯmYX#T,L))c]7IJ&. Y +f 6vl.vyͿ鲴ㆎ@mc222^l[cdߧ[Vcp1ny#6c}` em᡹l[C81^Q`gҺ -cuW7)DBkŎzqVf몾i6fC8 ZtY*oϝabip8^*84)惩-oqш8}'pF`Cz>Ȳ7-0lW4*@#GBJ na1 r-ޯ?KIqi^d pBtԠUs"A<S"l|o?^9"s.Ӽ>AxBW'[yPc3c.f;r(,߂c}~NKk5ԴuP߶T^{={ko/wXo_f!i<++Bz$ނ8%d7^ 4d^'s'X0J$r}bβj 6P a\-)il9OYb7lU}\{Dk8qn7U&jFĞcӒ:eQg&8a-u,/ +g=>-:k0>ꍷk-2 J=/vo^7}oeC$9F\xㇵ7Ɣ_17:JHkV!v: 23wigfM=Cˍ2o;pHTe»}{Knzm?ވ+|[[tPUv }G.>uyZۖa [vM=%"Xz߄h졇vd(9'î!;9" Ji_w 7y iM7.@KfA2Z{('ު¾0SﺰlfEo__Ox͂j G Ǻ7I!];gߏH~Wܯ2.F] KIyiCʵtYKϋ=] cJ{m >S,+}|.59~|۵oh*gOouy|~ӲJe,sǙzɍAvs:L!S’Y=5ԍM[2]Ѵ]÷8WI%ui4@^ܻ4ӢKwa{t[Ckmۏ8=cj P:y9,ϵfHmb~lH^ N8^,6lb= DF䗓z,#ض.n["c?%&?PW#C&w&2F~z.pX!g{]lr(!VAض;$[O14LɎ)I!~Nuhf6kLՐ+BvhTjrZ(}nWh!MJG׏kPm)vviueK~9+(T~3u{Ԕ&3TF>Xn&%>f#4cC4`9lJfG}LJ{ֺilET6S H5XmFiX9 +:94S~Yj6Y6n'dž/#92&owJLobNrw56@:DI^W?v'N韧s>8!z+,{/|0|;jhI&ofJln36ZC;ǝ{w^[چqGb:Ȯ~LBl7/}]|"1Rb\.s~$!q%M.wạ#.Aq9LXd+1 IE~\Fȥ<8tM 80qҲGaA{Զ06ө~v;]Cm,?oANwZbс{A'J[MZQS).ݪZgUKQaܕ0Jܵqע%0sꫣ޲ŒYűwro+m?7箧yXwAbLw #3TVg_߂mOo#0jPZɾ"Rk%LSne]0m0T}ޭ:Ѯ4\sf*$KWY^XRFI?A +endstream +endobj +856 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 855 0 R +/Resources 4 0 R +/Annots [ 857 0 R 858 0 R ] +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +857 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 797.673057 101.433071 778.923057 ] +/BS << +/W 0 +>> +/Dest (cb228-1) +>> +endobj +858 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 446.531993 101.433071 427.781993 ] +/BS << +/W 0 +>> +/Dest (cb231-1) +>> +endobj +859 0 obj +<< +/Filter /FlateDecode +/Length 4384 +>> +stream +x\[, ~_1J NLy3>k;Tun58gٳR]T*T=lɹϧ_N֤`\T_YN&s8,J&Sgaw}aw}BPyI%h0p׳P Ǫpt`5,@ϓx%+O%jX.cT^1.ABX4,]*Ұ wY`AD#DjuX pyCа` ߌ0]W-9kX%aH!9 "հ wދkkiʃ,k*>dXx]km*D஄M b@@,kX3R԰  R> ɂY*pb XB%kX`r4QECBIY=l9}z'RXq_݀+SUXI.U !k !`q=t!n˹p[!ԕo.wrVGç}Qt$`[XHj"@!4 kfDIvy=(UE% +ZP?X4 NuKGɍЭw?zWL?Xo_vW~Zw^k&۹e:TmXZimJ< ~\S}y]89B \x9HȨ< Is\÷H +չu.-k[S" +#\g(cG[íH jy29i3: +ͮn &G>*= vyK6no~}7\8YI}]^kof]\$aHdg[wlxXm~w?)a&y8m܏wwv&o/kg{*a ;|:& 'js ڣG͇q=U3߇xy3{WlRzߌc|0V}w!=:>XJLNS"t\gTof|0rа.tAnwq=n8=wΑ$ɡqoa^)LJ[Ryox_\-koMrIdae!'cؽ-r"ˍi>$T+WPǖB^8aYg}%4puг~#W y:>P5o_}?߬aM!(,29-{@ackYTѬn&yz]iP'y܀;|oQgQ\I@þ߇=i{o&!^`Ee+ A,^lE5Y'WyĦMx͂0t,^Ϸ}]>5!P ıԴI/'k aCj .D a\VKh E>,,,|^[rOްxZik֊1,YR5?x<`V:ě}/v{tuEKWu:įp9j?n>IBKp[;p{¶;v7NqzF^÷ R> <aqmrCMU~][ÖŒ]hqC>#}^/{q3ͼZ>c]mCks;r0-a^@:ud9I y P8PrDZ7'e}lwB}y +};JLUs=d^'55s e&g˭۱=Cu<@ồKU{?#\W9JV=#vCg8gt~8Gw﫩Gk6ΆƎ:7g^q#OΒ܄C ꥔17L4coޭ֯VGEyrɍ[+v=rc!{"(a +g\Nᨩ,{apj+nTU.6O1e\J0|v*b +!5SZBF#)JS9kH']U;?ӳ~/wr}?#%a[R_,$O' +yd27B\?>5`eߒǑ=m5?n< ֈk +=;eָ,_8 W"E'yn;oىD;M7ȞTyͽ մhtjh-p't9)m?횼,s3$Jk.':Z;zwɻ]>dxd1&do{M!@a>Ur֎N}>p2-^N?j<-W9Y;m?O|;qo;Zd"Fn3=O?.YCOxͣZsgb:Ⱦn}k&\!^vZ%i +rpFH>OCJ.#.Aq9MXd1 IE~dbJ#Gn R`/͓<ЯvU<@;i Nz=P}9P&ߣ o.fyt[yRnͤ-PtLqRWU-S Oü+]aޕxh+㶈׳% P9ѲYŹwhIvsh{C82S;CWk|g;uZzxZ^=T@!,߁8OG#/sx"J晰lrUO`یi5zb%LQЀ9{8 ߅VŭB2I(_}HfƦd*(ysR +endstream +endobj +860 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 859 0 R +/Resources 4 0 R +/Annots [ 861 0 R ] +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +861 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 797.673057 101.433071 778.923057 ] +/BS << +/W 0 +>> +/Dest (cb234-1) +>> +endobj +862 0 obj +<< +/Filter /FlateDecode +/Length 4884 +>> +stream +x][~_D l$@"Xa;N8y?եKU/3g;.R"?R*9ٙK|<}:|>Xꡲr2(TOt_ǹ_ޔ = +ɤ +e)LJ>DKX#$g%%12 N1|r +!),g!֪<b+<Lң\|Qy <#21 Nc Y1d : KpVA4^,ɪ)ñ$ :Ӏ,d 5$ Y ְXaw=a  w!AO V{NcL $&ŧqeCYKI\Ql(6'(%8flCQN] h=}_s%U +ʮIwȋNۤoǜ/ F<+beߜ}sg:dK9/87xcBCMy(p7YS|dS˲EEew.A_:rnV򵞢.+߼?z` \_ $w?61lƒū̚-ichm;wb @@׈Tmv9xfokzmcn>(ļcCa#6Hy]dkkuF,"־uZKr|qڷ]mLHqiAyZti6%愵2-*]KU45\@1$km'f;u$pW3XGLS!%Xrc3 +YxyU*y"uDX7Tr7?XfuI>b'ʼZ %\QQh.]U^e5{c= {p 'YN!ra~)Wj;;$ASJ&dmmLW1), ܒ:oϜr0 j2ӵߘ]g0{xILI0F̻ ˗9vxԍ1yDa+ߗ{WC^!I4fnL|)[Vۼ=H83yvX>6-jp<!3+c6)qOcQ%49OJ^tE_ ³%KKs>J 8Jz;`x29Al5}퇯\naDT<\9@p ˗B*)4E_*\cY._ՎKv_7hQ[jIܝ~n}nf#Nns(a%Su1 V59 ڵ]aM3\K?lBmKe*'zRgV}6+_C͎(k TΛIY2(Tt0yRo{a^.cU;\A}XFMuD|OX]@on̮7pڀIz;HGnPW%<]pI +!Б`)[@wy; vixȶjc3[ѭf6]ql+*{IbFq_49nj/nh,r+'> ;c䈹/v&Q.OG ~`r:)0L֢GڟdQR:jiqBi|F"WocdC2( e lbBz^JʕrTa)N𧃤1]otZY!#w|-DpMI +.\BxV-2"jF'/d6Zb%&3AˎGśdGv"=N &Ѻ vgչi[YF<-mO*UnlM5&JVOyRHmM3&Y {)A-L^*i~2LrWB;MGؓ\=qMS}mܑE + {"Mvr^5*fMO*B +Immit"k%Q5 8N.RY_.[n/5N*4:xɮ[]' 5t 9tS~Zj&sA;fr3.oowF<[~n $|zgqPOܼ{iit9OKRtKo4Pǁlp>qo;Z+|'Y!o6(58<쭤:w&Épxh'2_:7#` ݐ\zפ<3nG#$&$EGHTsx:,Cbs']"?S!5)G<avn("Hi~f7wZV(Bؽxcvhԧ&x:c\?CGyh;4ۢ'95FjF'jՖami .@[LMTkj#ΪR)ﳦ<+]a++׳% P9QYŹv(I}h{C82S;]K|GTB+ԝerS[lrw> +endobj +864 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 393.004475 101.433071 374.254475 ] +/BS << +/W 0 +>> +/Dest (cb237-1) +>> +endobj +865 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 374.057453 99.933071 355.307453 ] +/BS << +/W 0 +>> +/Dest (cb237-2) +>> +endobj +866 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 355.110432 99.933071 336.360432 ] +/BS << +/W 0 +>> +/Dest (cb237-3) +>> +endobj +867 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 336.163410 99.933071 317.413410 ] +/BS << +/W 0 +>> +/Dest (cb237-4) +>> +endobj +868 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 317.216389 99.933071 298.466389 ] +/BS << +/W 0 +>> +/Dest (cb237-5) +>> +endobj +869 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 298.269367 99.933071 279.519367 ] +/BS << +/W 0 +>> +/Dest (cb237-6) +>> +endobj +870 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 279.322346 99.933071 260.572346 ] +/BS << +/W 0 +>> +/Dest (cb237-7) +>> +endobj +871 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 260.375324 99.933071 241.625324 ] +/BS << +/W 0 +>> +/Dest (cb237-8) +>> +endobj +872 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 241.428303 99.933071 222.678303 ] +/BS << +/W 0 +>> +/Dest (cb237-9) +>> +endobj +873 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 222.481281 99.933071 203.731281 ] +/BS << +/W 0 +>> +/Dest (cb237-10) +>> +endobj +874 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 203.534260 99.933071 184.784260 ] +/BS << +/W 0 +>> +/Dest (cb237-11) +>> +endobj +875 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 165.640217 99.933071 146.890217 ] +/BS << +/W 0 +>> +/Dest (cb237-12) +>> +endobj +876 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 127.746174 99.933071 108.996174 ] +/BS << +/W 0 +>> +/Dest (cb237-13) +>> +endobj +877 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 108.799152 99.933071 90.049152 ] +/BS << +/W 0 +>> +/Dest (cb237-14) +>> +endobj +878 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 89.852131 99.933071 71.102131 ] +/BS << +/W 0 +>> +/Dest (cb237-15) +>> +endobj +879 0 obj +<< +/Filter /FlateDecode +/Length 3166 +>> +stream +x͛[$ S̳aRIu`Cpa &M8X!_?t]zF綻&ٙV~g+?U9JӇm%{tf\ R,|?Wk_}ZˑMJ-jPJ)SvTn5O3RѨ3`XUV`F, +fpKQ1DDG)d8={!+qN$䢜ӵsN)lI%ؒX<lQe"YXx["]-` ebO$VU]]jH)*B9$Ħ:( R..–L:ؒآtDR3[n!aO`KE]l{P0qc@,%hA V Zz-E< ĖV<1t=#0lQ\W[<@@[Ex,Vo8{,fqfX @.By,0WNbJc5{,pWc"` cx,PpWw1j@fk% +J@X0c1܍wrpI~:[XXcAwas4h \1|.3o97 B%  |V`|=$p+w96X] c!0XBյ@V *HJ@ 8P<dpǂ "quEjX:L[gN#Y A +Oc=KTǂ ڲ dcAZ)cAnR>WǂB$ -uhҪǂa豠D؂c2ٷ=-w,Ԧ걠r ׮BcA)BŤXK< X=[0U<`) +5"3t [d{,@ ǂa A5V23y, +$\lY=TRE&i 0>N fW[$ǂZm7ǂZaKiⱠlA籠Q&{,ơ#Xиbc7ǂ&&xμmmY<4- )-H3o$7A'OaKl걠li#jLEf=MdޞTj;ЛGb*Gr `&v5XVdr E>7L>GTPfb70-sb@o|NN#EmM0Uؚ퀇f! :fDiHC73K1v_h2,6vgc$guJ0Ql _k =>gtg|~[!wot7OZ'F +gTl~@`Ҏ"ے],qcjF)iv2Coޜq *<~7Coޝ~ʶ7>2,TCkր"лׯxkOo>)(v^~V܆:W OTZB*ßpE?;^ll laϪǛ$Q%Y/yi' v%Yvw2u` Jgrȉy)՗SYFr7@TkBڦ}aG,d}w.eR//!|=4uC?6ј:o@aÑ(|*{M$1nP%D^(&9wݕ=wPKO!Z%}GI봇|phh_t^uC={l '=ϫ)IAy$j32"=RzDl5cYt~nʳ~AIǢnDT [r ݣnvFbכj8{¶G6JWox{t%IJfZ ,nztdvh^D;sQ|E`GujxoW|Aᖓ%-HK9=H.ˇɷU=_~6X}gأ=,9q}Y8W9^0JoM=HC+U^6O4-T/o8&_LpQG3&ܿhK;LuIL'ڿѸͱטXԫYn vvf[=P _aJfg ΊOYh[!-MqnSL:>tܘI|lx9΀m2d$9sjBk.:کv١ӾP \@F]6Z,ݦWn^*#_WOuT:*W-He͒#,TgI۾w_mm4$̔ftS1sR+ ,ElB5g'lؒȋ<~Vr{"J֞0AEo=iT&Z#^5UIvb[E{k=ڎeH5 +endstream +endobj +880 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 879 0 R +/Resources 4 0 R +/Annots [ 881 0 R 882 0 R 883 0 R 884 0 R 885 0 R ] +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +881 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 799.173057 99.933071 780.423057 ] +/BS << +/W 0 +>> +/Dest (cb237-16) +>> +endobj +882 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 780.226036 99.933071 761.476036 ] +/BS << +/W 0 +>> +/Dest (cb237-17) +>> +endobj +883 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 761.279014 99.933071 742.529014 ] +/BS << +/W 0 +>> +/Dest (cb237-18) +>> +endobj +884 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 742.331993 99.933071 723.581993 ] +/BS << +/W 0 +>> +/Dest (cb237-19) +>> +endobj +885 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 681.384971 101.433071 662.634971 ] +/BS << +/W 0 +>> +/Dest (cb238-1) +>> +endobj +886 0 obj +<< +/Filter /FlateDecode +/Length 3336 +>> +stream +x[K$ 賁%zH8!FzsGUIbusfgvٙ.zPRMJɻ<Bg%Rh!PqXzŸ~=_6]k9tGa1RK`5| +%lrpbk b‘YbpbA  +1K{VX#T V#DM ^H.Crm^(\8`9jK%fo S-/CMd+%B;U&kR&kK[\mɃGar1xdɃ,|xY?8jxk&O%caY Y|-BXRȂؒ܊x1X@@$P, 5y[X@D%2YX@@[XX@[-XX@[![X@\-, . ,, `.;-, .ܽXX@@]J9ZX@]-, .q0p-Md[Xtt*YX] +XX]Q`؊z g  B+>H0p7&Lݘ|c?dka=x,,@g."- wCC3|hBm‚ g 7Zv!faA"P "j w!R[N`&9&Pb jݬ +Y6eY,,h 㳰$M0PߌlaR‚FMl meY,,h YRY45oIo,^,-goV f`<,qp75 "3&2[ko ?Uv<1ބjx`/U1 w># s($܂=ǪFT~;O>=|OƏF'Lccj8A!"͊|"3!`z[b?I-:둎`t5Q[ɦ5EYҪ;D^.UQפPK;~/uK&N|0Ɓ&.%j!ZceUƿ*=*kr7Y5bPA#Yϣ.Z i:a-s qŰjPU6TQ;DmCٴI]^C-Xzs%Ueڙs`;NvЛ:o+8XX+`ٯ|B3e)#xs秜ޞ YTƎȑ㟰|p7އ߀"g)m !o-zhO~w n]w|ެ"8eT~&|OK{;LEٟuVDI>C]n[WqoyXlӼ㲍7e<Sw7ކw/&{|scW]4i/}*08'e ևqO&9ЖK[|OA?&Ǯ+.K'iNqtC6<=܌z_6{rW~;O?LuYٜ%s2>]x}-< + G6=sՋ躷tضom+ӆLi{/ =!%ZO#dZ +ǪMK%'~=P+ }?/CN_ }(X,9ޛGf.˦,ސ.{Ѿ{RVVtÍwްF! M.D~uqk' Jߕ^N6Ag_f,)njRr~\xiFڧLJ#$6 l2{޻yu{6CuW'Ɂ"{W-r!OGSuyZuA-ZΧ(|jRjhMj5\'$;?f,U?CYzr2Re֪i_,S:o.*,#,9\k_;儜vZX0=M$$WeQKkܝTIIUZoNQI'갥eU +L +tgULK HM8ôIX~9MSK4]>A{.ry2W=YM ye~5Kw5? >wW`VOBB]g9Կk/hm> ȗ mE>P]͚72> +endobj +888 0 obj +<< +/Filter /FlateDecode +/Length 3182 +>> +stream +x[MdadIUc! 8d0!x1؄8R_}ƞ" x=+JUG%]u8{ +W5Z[z?}8yWO/qw_I൸Z)K>_Ov/~=}τstLJ-'"j D+%LD98t5&hGQjb"1V"PCy5KP&BDC2(U%YH.bpب@Jd2sL%5,AT=AZ,C Z +tN# +]]ʠ>0YZU>ʢO.Ud!*CDar!]OʠKLIJ$ZEx LKE ` ]|ɌAxq@,iH VqZ29-E ĖV@) FޗƸ wmd\2.Ypd\VJf\ )eĸx EeE2.@b횴ʸD"12.( e`\. ܅ Sd\&s2.()c=2Y5UBa\P:LŁqR"K dA]* 03Y.H Ӆ,5Bd jXefϸ34֒05ufe\P5;K LSRtM0.b;1.M,s HRF[nVB#7 2>ggi28d3.hbq֛qA&x$nd[oMKBBeIS#q NzdĦ Zζެ f`n T[o]M d]dqnDfě +3.tzVe +3 _[#Met*mE>p}ڷ2ܡ`ά:~9V1ymO_{?w_k,HA0O0k_WdO`M 6RC0X ֎o!76d',"}vPW ^=njjYracYR0Y! psX.%J#P`C2kqf.p*{E(·|C;ixl.ؼB|rU~ +vpL}hӫefK_"Ru9?^{ٝ6ۿK΂/O9?^|B +o= q/|/GRg%^wx>|\|6uŶ4ϯߟ͟w|} O/G|Sq}_;ÿgڳ=k"*AuܪF)ޚMuoͧtwBϳ@-N4ۡRG~*|5:] O`3#]E3mW뼞mCvʽ3M}8/7m>́cOᦹ>-טo;,nhaj0WBYrnz |'mQzh9tK]!??ƠGHn1Ct#9k̍=:ovp"1vuYYz G,3:ˡC^z(>{Gn8돯-~ПS+m;حE>G6o@[QDGxb+Qe&0D؎NY;!o#0YwvÍ'uz{on^ԟ {*}H;I\]vi^z0)tÆ3OfDC*v\u9 .|k7 Xu/~֥a#X$5H1iXbc鴸§z;([~Bt9L{7H5AKu؜v RHwC7m8~V'yaz݄†}&WJ;xfGc9lf Γ;;\; 2n>?K)l_G# ~v \h\AfS梤lsKmܳp/|ױuǣl'Lf'믄dgl`ԍ*7y<ƍu=a2hLˆ eG+D淎͚o2:fĎa pwudMJY5Cq{7{lK;5!FM݇BfK#VL4W=ދR`o5썶;}ҟ^yC'OcN|;9eYl'3YR }tCz(Fu`wb0?Yc:aJTfNПvK}T>*;f-Leί3URض޷=f ]䕙Xz(Y9CYwBB]0Qk(ל!j‹mk䓟O{K"n(W +AeV(yڥO3_o +м-T%վP3/Fm{r +endstream +endobj +889 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 888 0 R +/Resources 4 0 R +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +890 0 obj +<< +/Filter /FlateDecode +/Length 4326 +>> +stream +x\K$ ׯ˂l 1>Lbz >ST*BŇiz:CP(ORTe;jtVr<>)_jt[XLr-tLPLiɗ||:MSv:{ћֲw'b`J-&.(L)>e[X#U Fl5i, +B {V k(,g!֪<bc ^d25w&+5^&xSxfK%ƃ,1Ԣ2d 9XY|-A⡝*duUYlN`R1y&jV9gl3a ZdP Ǫpt`5,@ 0HK4,H]ZW[ְ w]f\˹iXTAÂ|V kX. Rk IÂbAH,Yb 6-aA%xaA)LsȾ\ BdLm1jXPRF=jM-kWsEÂRA(v2NaPaE J,p L5,H5,@ sTY jXP}EШ%j-ՙUÂTQ *M ʃ, ?Tl' PD KHQÂZ=UÂZ!^&՛EÂf! 2> ͆d YaAs/ެ`QoSLfѰEȒ<Ȃ)~ LB tx5,h9SY5,@zz$P,qp75 "3zV[o*tf'Л[# E*ꙁR@LT5`:X/Ä<0Y --"QsnBد?}@=769)`v +Ay6 ~jщ #:8ا5U]t1C ID(Pn)s:IsI+yԹ9Ԏ'l9&@ ѭHwa{vͭv.%zVciM֦jJnbE7"*pk2UW;ʄ&ymB䇍eL(l*&du„*zo)OkI4РEϩv4⸔ӿBYϺC^t277{sߙeW;O,W,7{s7T'/ox3ǻ=ܗ??@mX!}χo?翠~|xwֺ_:o-o߈3?7s;𛗿7awv& +YbCC8*uw)M"I=Kx]:x$8:?K&H",J;4ag?ȴ/->mu9BeG;.^~:kȋYM1X0Cc qrj2~g_cI\ee;!llMr{BvYN>"D಴OﷲmqSӶ]w_q!/㞦ײN{}XcuV4sXޔQ,Mt'QJɄ}@hƞ5-HVέU=坞nч'ۘY?lm^41Js|ΥXi:&viLR3=PeӢ{5ǪY9Ű{3uI_@>ْ۟Эdt;W /C(c;t}ROΕW{n>w ^]_V_+[c[QVu͞6kF˜@Ԧlp돎}]wc@x2j#>Lܾ.~7e:8~mm=VsUg:GQ_Ր^g]f5csu]ӤrMwyû|nra2ob{j\CmUS64/ko| >㴍CƁkLok!4 ic=Or9-;f_e϶ykw ?~\3:Ym$֒XKgE^ +p, +;&r DŸt@ؖ\YcV;t bsNO'z̓թ8`(=q읧o:&%'_d?}&պKr,S$UO/ucih,Į3m2@ϖ3 +rOKv +bn}dy= +=T'q.Us_T&iÖR<kA B6{ c}ApG|gy|kXXِ׸B:~hiT꼵(CWh'qzWrMDj쵭nx:8?_x+?Fp[WdcF2rmGF^u u+iP0]LBIXQЛy1';54 +Hrk"FۖINŀ ,ƈemjbRNrWܕKrR P\9rɪв{P;h{ˈS<3S;oK/¿p֋IѕCU9C"m*F祯3a;L G/Ex=C yۚUt. E}.ghҧ^(nb/> +endobj +892 0 obj +<< +/Filter /FlateDecode +/Length 4317 +>> +stream +x\K$ȳ%X 0vჁ6 :Ƣf2Pd2˞ `PG(2=~ ppԜ|ɻ,>Ysvj*rI ]s:_w'5 <+咳U 9J2X)8b bH 1Xc,@ U!F́-U +rɃ ,^ .-KW]JIc"'Exd %&Ēej +duY|UrR ɫ=/zqR lLv)e!KRMddaX‚A@Ƀ,֮R-Y(F r,,,@Dj@mb Y)ZX$s2[X%!V-^d+laAȥ\4\b‚\! pEDD/@ / J,-,( LxhPcyfpr*ZXmLHi `&ba$HsLd!a Jn J,G `of ,,,>9 -,罅5䛑-,|XXP#DgJIlaAe"QLdAT} LB tx Ԕ4, @, 7Y.J2Y<."da.o"r{f8Y1^o5K o͚jd &2˒̪*eE%s%MVl Ft?rZN~=N_~*e'&H7R rm8F0A3a4 !Q!r] ťB#lE*s@0y),%[7y'YW%Ѹ(g%!膯Qn`w<$K7ywYi!OBX/yM!JG! B5~G!7N{:O!O_~/Xfx\w8#a|D:]yzy5?\:kU#YҴn6s ڧp6S7w*3]tո1G5j 󝢥zqj~Wj3>e d$.j˲+R}.C?,o/HhQƆ /Nǜ.}sqH!?R@a]*qC煞ה B`̇MAYc`j='MH + v{,*w}cʲuWnqf^,ه,~$W)z]cz)"%!j|E'>ϜCTmX_ajօik͟;_fd1Cʜc/;'.L-=ؚ-jzy<`uARFq1G,Hz9] 8rh'*}>m +ABsڏ ݽXduK VsyÈUE0ɼ/U(~l'r`2?nn/JAJ)qcɎ_I1o̢&7CK[5g*3Nfz0'aZS䧛o\Z< [|#}|~\t8 ^ .|6;꟢n;O79oCznI%lYܜRVCzKI DUV>=ўGc}>LEaC_k&{8VK>Fݍ\8ajs3ϲ#Let<Y>{ R܅ zCR_17L+ 9 +zQ'\ˤx u0PѻXL5}!_ y&13;q_)RZ<OcwO$.1ľil5T.F/I.jCz-0~7}rHv5rñ.(P0h~#N{g;[+FoɏvۑxqownJzS MG.!m#rǺPJXҎE}/z}~i'~쫾pjIנ=<^R+^Kݟji՗n&m}2B߷>G@Ӓ\h=8Micz.ql7X PI/GNVblVJfGWb͈wGp?j}!]_Gm;9Bs f!C$ZYߵ1/m#k@|= M8u%pn;eG!m|^ Q]CgfZ)ŏӻe{ 3A=|l?R{G,=3ΛtYO2p~gG+$/i zm狥m:Nz8ro`PXƑN`#m?ǮFwwsJm:ADzޅ@R-WiuA\:qOD'iա´mr>"^Nn/&U>TnFPuP c d}haUӷG Gl&chU֮k~e5u*ChfJi}fKkd7x9 _Nֲbnz3q^V*EUؚWAUi&ϮW$7y갥ym;+> +endobj +894 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 766.923057 101.433071 748.173057 ] +/BS << +/W 0 +>> +/Dest (cb241-1) +>> +endobj +895 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 747.976036 99.933071 729.226036 ] +/BS << +/W 0 +>> +/Dest (cb241-2) +>> +endobj +896 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 729.029014 99.933071 710.279014 ] +/BS << +/W 0 +>> +/Dest (cb241-3) +>> +endobj +897 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 710.081993 99.933071 691.331993 ] +/BS << +/W 0 +>> +/Dest (cb241-4) +>> +endobj +898 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 691.134971 99.933071 672.384971 ] +/BS << +/W 0 +>> +/Dest (cb241-5) +>> +endobj +899 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 672.187950 99.933071 653.437950 ] +/BS << +/W 0 +>> +/Dest (cb241-6) +>> +endobj +900 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 653.240928 99.933071 634.490928 ] +/BS << +/W 0 +>> +/Dest (cb241-7) +>> +endobj +901 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 634.293907 99.933071 615.543907 ] +/BS << +/W 0 +>> +/Dest (cb241-8) +>> +endobj +902 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 615.346885 99.933071 596.596885 ] +/BS << +/W 0 +>> +/Dest (cb241-9) +>> +endobj +903 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 596.399864 99.933071 577.649864 ] +/BS << +/W 0 +>> +/Dest (cb241-10) +>> +endobj +904 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 577.452842 99.933071 558.702842 ] +/BS << +/W 0 +>> +/Dest (cb241-11) +>> +endobj +905 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 520.611778 99.933071 501.861778 ] +/BS << +/W 0 +>> +/Dest (cb241-12) +>> +endobj +906 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 501.664756 99.933071 482.914756 ] +/BS << +/W 0 +>> +/Dest (cb241-13) +>> +endobj +907 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 482.717735 99.933071 463.967735 ] +/BS << +/W 0 +>> +/Dest (cb241-14) +>> +endobj +908 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 463.770714 99.933071 445.020714 ] +/BS << +/W 0 +>> +/Dest (cb241-15) +>> +endobj +909 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 444.823692 99.933071 426.073692 ] +/BS << +/W 0 +>> +/Dest (cb241-16) +>> +endobj +910 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 425.876671 99.933071 407.126671 ] +/BS << +/W 0 +>> +/Dest (cb241-17) +>> +endobj +911 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 406.929649 99.933071 388.179649 ] +/BS << +/W 0 +>> +/Dest (cb241-18) +>> +endobj +912 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 345.982628 101.433071 327.232628 ] +/BS << +/W 0 +>> +/Dest (cb242-1) +>> +endobj +913 0 obj +<< +/Title (CIS v1.6 Benchmark - Self-Assessment Guide - Rancher v2.6) +/Dest [ 6 0 R /XYZ 85.009843 502.818898 0 ] +/Count 141 +/First 914 0 R +/Last 1050 0 R +/Parent 1055 0 R +>> +endobj +914 0 obj +<< +/Title (CIS v1.6 Kubernetes Benchmark - Rancher v2.6 with Kubernetes v1.18 to v1.21) +/Dest [ 304 0 R /XYZ 84.259843 784.970079 0 ] +/Count 2 +/First 915 0 R +/Last 916 0 R +/Parent 913 0 R +/Next 917 0 R +>> +endobj +915 0 obj +<< +/Title (Overview) +/Dest [ 304 0 R /XYZ 84.259843 704.822079 0 ] +/Count 0 +/Parent 914 0 R +/Next 916 0 R +>> +endobj +916 0 obj +<< +/Title (Testing controls methodology) +/Dest [ 304 0 R /XYZ 84.259843 166.076079 0 ] +/Count 0 +/Prev 915 0 R +/Parent 914 0 R +>> +endobj +917 0 obj +<< +/Title (Controls) +/Dest [ 309 0 R /XYZ 84.259843 612.381496 0 ] +/Count 0 +/Prev 914 0 R +/Parent 913 0 R +/Next 918 0 R +>> +endobj +918 0 obj +<< +/Title (1.1 Master Node Configuration Files) +/Dest [ 313 0 R /XYZ 41.740157 784.430079 0 ] +/Count 21 +/Prev 917 0 R +/First 919 0 R +/Last 939 0 R +/Parent 913 0 R +/Next 940 0 R +>> +endobj +919 0 obj +<< +/Title (1.1.1 Ensure that the API server pod specification file permissions are set to 644 or more restrictive \(Automated\)) +/Dest [ 313 0 R /XYZ 84.259843 684.651496 0 ] +/Count 0 +/Parent 918 0 R +/Next 920 0 R +>> +endobj +920 0 obj +<< +/Title (1.1.2 Ensure that the API server pod specification file ownership is set to root:root \(Automated\)) +/Dest [ 313 0 R /XYZ 84.259843 553.851496 0 ] +/Count 0 +/Prev 919 0 R +/Parent 918 0 R +/Next 921 0 R +>> +endobj +921 0 obj +<< +/Title (1.1.3 Ensure that the controller manager pod specification file permissions are set to 644 or more restrictive \(Automated\)) +/Dest [ 313 0 R /XYZ 84.259843 423.051496 0 ] +/Count 0 +/Prev 920 0 R +/Parent 918 0 R +/Next 922 0 R +>> +endobj +922 0 obj +<< +/Title (1.1.4 Ensure that the controller manager pod specification file ownership is set to root:root \(Automated\)) +/Dest [ 313 0 R /XYZ 84.259843 277.251496 0 ] +/Count 0 +/Prev 921 0 R +/Parent 918 0 R +/Next 923 0 R +>> +endobj +923 0 obj +<< +/Title (1.1.5 Ensure that the scheduler pod specification file permissions are set to 644 or more restrictive \(Automated\)) +/Dest [ 313 0 R /XYZ 84.259843 146.451496 0 ] +/Count 0 +/Prev 922 0 R +/Parent 918 0 R +/Next 924 0 R +>> +endobj +924 0 obj +<< +/Title (1.1.6 Ensure that the scheduler pod specification file ownership is set to root:root \(Automated\)) +/Dest [ 315 0 R /XYZ 84.259843 739.970079 0 ] +/Count 0 +/Prev 923 0 R +/Parent 918 0 R +/Next 925 0 R +>> +endobj +925 0 obj +<< +/Title (1.1.7 Ensure that the etcd pod specification file permissions are set to 644 or more restrictive \(Automated\)) +/Dest [ 315 0 R /XYZ 84.259843 609.170079 0 ] +/Count 0 +/Prev 924 0 R +/Parent 918 0 R +/Next 926 0 R +>> +endobj +926 0 obj +<< +/Title (1.1.8 Ensure that the etcd pod specification file ownership is set to root:root \(Automated\)) +/Dest [ 315 0 R /XYZ 84.259843 478.370079 0 ] +/Count 0 +/Prev 925 0 R +/Parent 918 0 R +/Next 927 0 R +>> +endobj +927 0 obj +<< +/Title (1.1.9 Ensure that the Container Network Interface file permissions are set to 644 or more restrictive \(Manual\)) +/Dest [ 315 0 R /XYZ 84.259843 347.570079 0 ] +/Count 0 +/Prev 926 0 R +/Parent 918 0 R +/Next 928 0 R +>> +endobj +928 0 obj +<< +/Title (1.1.10 Ensure that the Container Network Interface file ownership is set to root:root \(Manual\)) +/Dest [ 315 0 R /XYZ 84.259843 170.823057 0 ] +/Count 0 +/Prev 927 0 R +/Parent 918 0 R +/Next 929 0 R +>> +endobj +929 0 obj +<< +/Title (1.1.11 Ensure that the etcd data directory permissions are set to 700 or more restrictive \(Automated\)) +/Dest [ 318 0 R /XYZ 84.259843 694.023057 0 ] +/Count 0 +/Prev 928 0 R +/Parent 918 0 R +/Next 930 0 R +>> +endobj +930 0 obj +<< +/Title (1.1.12 Ensure that the etcd data directory ownership is set to etcd:etcd \(Automated\)) +/Dest [ 318 0 R /XYZ 84.259843 365.776036 0 ] +/Count 0 +/Prev 929 0 R +/Parent 918 0 R +/Next 931 0 R +>> +endobj +931 0 obj +<< +/Title (1.1.13 Ensure that the admin.conf file permissions are set to 644 or more restrictive \(Automated\)) +/Dest [ 323 0 R /XYZ 84.259843 702.470079 0 ] +/Count 0 +/Prev 930 0 R +/Parent 918 0 R +/Next 932 0 R +>> +endobj +932 0 obj +<< +/Title (1.1.14 Ensure that the admin.conf file ownership is set to root:root \(Automated\)) +/Dest [ 323 0 R /XYZ 84.259843 586.670079 0 ] +/Count 0 +/Prev 931 0 R +/Parent 918 0 R +/Next 933 0 R +>> +endobj +933 0 obj +<< +/Title (1.1.15 Ensure that the scheduler.conf file permissions are set to 644 or more restrictive \(Automated\)) +/Dest [ 323 0 R /XYZ 84.259843 470.870079 0 ] +/Count 0 +/Prev 932 0 R +/Parent 918 0 R +/Next 934 0 R +>> +endobj +934 0 obj +<< +/Title (1.1.16 Ensure that the scheduler.conf file ownership is set to root:root \(Automated\)) +/Dest [ 323 0 R /XYZ 84.259843 340.070079 0 ] +/Count 0 +/Prev 933 0 R +/Parent 918 0 R +/Next 935 0 R +>> +endobj +935 0 obj +<< +/Title (1.1.17 Ensure that the controller-manager.conf file permissions are set to 644 or more restrictive \(Automated\)) +/Dest [ 323 0 R /XYZ 84.259843 209.270079 0 ] +/Count 0 +/Prev 934 0 R +/Parent 918 0 R +/Next 936 0 R +>> +endobj +936 0 obj +<< +/Title (1.1.18 Ensure that the controller-manager.conf file ownership is set to root:root \(Automated\)) +/Dest [ 325 0 R /XYZ 84.259843 799.370079 0 ] +/Count 0 +/Prev 935 0 R +/Parent 918 0 R +/Next 937 0 R +>> +endobj +937 0 obj +<< +/Title (1.1.19 Ensure that the Kubernetes PKI directory and file ownership is set to root:root \(Automated\)) +/Dest [ 325 0 R /XYZ 84.259843 668.570079 0 ] +/Count 0 +/Prev 936 0 R +/Parent 918 0 R +/Next 938 0 R +>> +endobj +938 0 obj +<< +/Title (1.1.20 Ensure that the Kubernetes PKI certificate file permissions are set to 644 or more restrictive \(Automated\)) +/Dest [ 349 0 R /XYZ 84.259843 184.188585 0 ] +/Count 0 +/Prev 937 0 R +/Parent 918 0 R +/Next 939 0 R +>> +endobj +939 0 obj +<< +/Title (1.1.21 Ensure that the Kubernetes PKI key file permissions are set to 600 \(Automated\)) +/Dest [ 440 0 R /XYZ 84.259843 702.470079 0 ] +/Count 0 +/Prev 938 0 R +/Parent 918 0 R +>> +endobj +940 0 obj +<< +/Title (1.2 API Server) +/Dest [ 508 0 R /XYZ 41.740157 784.430079 0 ] +/Count 35 +/Prev 918 0 R +/First 941 0 R +/Last 975 0 R +/Parent 913 0 R +/Next 976 0 R +>> +endobj +941 0 obj +<< +/Title (1.2.1 Ensure that the --anonymous-auth argument is set to false \(Automated\)) +/Dest [ 508 0 R /XYZ 84.259843 720.651496 0 ] +/Count 0 +/Parent 940 0 R +/Next 942 0 R +>> +endobj +942 0 obj +<< +/Title (1.2.2 Ensure that the --basic-auth-file argument is not set \(Automated\)) +/Dest [ 511 0 R /XYZ 84.259843 295.970079 0 ] +/Count 0 +/Prev 941 0 R +/Parent 940 0 R +/Next 943 0 R +>> +endobj +943 0 obj +<< +/Title (1.2.3 Ensure that the --token-auth-file parameter is not set \(Automated\)) +/Dest [ 516 0 R /XYZ 84.259843 595.970079 0 ] +/Count 0 +/Prev 942 0 R +/Parent 940 0 R +/Next 944 0 R +>> +endobj +944 0 obj +<< +/Title (1.2.4 Ensure that the --kubelet-https argument is set to true \(Automated\)) +/Dest [ 519 0 R /XYZ 84.259843 164.720079 0 ] +/Count 0 +/Prev 943 0 R +/Parent 940 0 R +/Next 945 0 R +>> +endobj +945 0 obj +<< +/Title (1.2.5 Ensure that the --kubelet-client-certificate and --kubelet-client-key arguments are set as appropriate \(Automated\)) +/Dest [ 524 0 R /XYZ 84.259843 427.220079 0 ] +/Count 0 +/Prev 944 0 R +/Parent 940 0 R +/Next 946 0 R +>> +endobj +946 0 obj +<< +/Title (1.2.6 Ensure that the --kubelet-certificate-authority argument is set as appropriate \(Automated\)) +/Dest [ 529 0 R /XYZ 84.259843 652.220079 0 ] +/Count 0 +/Prev 945 0 R +/Parent 940 0 R +/Next 947 0 R +>> +endobj +947 0 obj +<< +/Title (1.2.7 Ensure that the --authorization-mode argument is not set to AlwaysAllow \(Automated\)) +/Dest [ 532 0 R /XYZ 84.259843 183.470079 0 ] +/Count 0 +/Prev 946 0 R +/Parent 940 0 R +/Next 948 0 R +>> +endobj +948 0 obj +<< +/Title (1.2.8 Ensure that the --authorization-mode argument includes Node \(Automated\)) +/Dest [ 537 0 R /XYZ 84.259843 464.720079 0 ] +/Count 0 +/Prev 947 0 R +/Parent 940 0 R +/Next 949 0 R +>> +endobj +949 0 obj +<< +/Title (1.2.9 Ensure that the --authorization-mode argument includes RBAC \(Automated\)) +/Dest [ 542 0 R /XYZ 84.259843 745.970079 0 ] +/Count 0 +/Prev 948 0 R +/Parent 940 0 R +/Next 950 0 R +>> +endobj +950 0 obj +<< +/Title (1.2.10 Ensure that the admission control plugin EventRateLimit is set \(Automated\)) +/Dest [ 545 0 R /XYZ 84.259843 314.720079 0 ] +/Count 0 +/Prev 949 0 R +/Parent 940 0 R +/Next 951 0 R +>> +endobj +951 0 obj +<< +/Title (1.2.11 Ensure that the admission control plugin AlwaysAdmit is not set \(Automated\)) +/Dest [ 550 0 R /XYZ 84.259843 520.970079 0 ] +/Count 0 +/Prev 950 0 R +/Parent 940 0 R +/Next 952 0 R +>> +endobj +952 0 obj +<< +/Title (1.2.12 Ensure that the admission control plugin AlwaysPullImages is set \(Manual\)) +/Dest [ 555 0 R /XYZ 84.259843 708.470079 0 ] +/Count 0 +/Prev 951 0 R +/Parent 940 0 R +/Next 953 0 R +>> +endobj +953 0 obj +<< +/Title (1.2.13 Ensure that the admission control plugin SecurityContextDeny is set if PodSecurityPolicy is not used \(Manual\)) +/Dest [ 555 0 R /XYZ 84.259843 501.723057 0 ] +/Count 0 +/Prev 952 0 R +/Parent 940 0 R +/Next 954 0 R +>> +endobj +954 0 obj +<< +/Title (1.2.14 Ensure that the admission control plugin ServiceAccount is set \(Automated\)) +/Dest [ 555 0 R /XYZ 84.259843 264.976036 0 ] +/Count 0 +/Prev 953 0 R +/Parent 940 0 R +/Next 955 0 R +>> +endobj +955 0 obj +<< +/Title (1.2.15 Ensure that the admission control plugin NamespaceLifecycle is set \(Automated\)) +/Dest [ 562 0 R /XYZ 84.259843 520.970079 0 ] +/Count 0 +/Prev 954 0 R +/Parent 940 0 R +/Next 956 0 R +>> +endobj +956 0 obj +<< +/Title (1.2.16 Ensure that the admission control plugin PodSecurityPolicy is set \(Automated\)) +/Dest [ 567 0 R /XYZ 84.259843 799.370079 0 ] +/Count 0 +/Prev 955 0 R +/Parent 940 0 R +/Next 957 0 R +>> +endobj +957 0 obj +<< +/Title (1.2.17 Ensure that the admission control plugin NodeRestriction is set \(Automated\)) +/Dest [ 570 0 R /XYZ 84.259843 258.470079 0 ] +/Count 0 +/Prev 956 0 R +/Parent 940 0 R +/Next 958 0 R +>> +endobj +958 0 obj +<< +/Title (1.2.18 Ensure that the --insecure-bind-address argument is not set \(Automated\)) +/Dest [ 575 0 R /XYZ 84.259843 427.220079 0 ] +/Count 0 +/Prev 957 0 R +/Parent 940 0 R +/Next 959 0 R +>> +endobj +959 0 obj +<< +/Title (1.2.19 Ensure that the --insecure-port argument is set to 0 \(Automated\)) +/Dest [ 580 0 R /XYZ 84.259843 727.220079 0 ] +/Count 0 +/Prev 958 0 R +/Parent 940 0 R +/Next 960 0 R +>> +endobj +960 0 obj +<< +/Title (1.2.20 Ensure that the --secure-port argument is not set to 0 \(Automated\)) +/Dest [ 583 0 R /XYZ 84.259843 314.720079 0 ] +/Count 0 +/Prev 959 0 R +/Parent 940 0 R +/Next 961 0 R +>> +endobj +961 0 obj +<< +/Title (1.2.21 Ensure that the --profiling argument is set to false \(Automated\)) +/Dest [ 588 0 R /XYZ 84.259843 595.970079 0 ] +/Count 0 +/Prev 960 0 R +/Parent 940 0 R +/Next 962 0 R +>> +endobj +962 0 obj +<< +/Title (1.2.22 Ensure that the --audit-log-path argument is set \(Automated\)) +/Dest [ 591 0 R /XYZ 84.259843 183.470079 0 ] +/Count 0 +/Prev 961 0 R +/Parent 940 0 R +/Next 963 0 R +>> +endobj +963 0 obj +<< +/Title (1.2.23 Ensure that the --audit-log-maxage argument is set to 30 or as appropriate \(Automated\)) +/Dest [ 596 0 R /XYZ 84.259843 445.970079 0 ] +/Count 0 +/Prev 962 0 R +/Parent 940 0 R +/Next 964 0 R +>> +endobj +964 0 obj +<< +/Title (1.2.24 Ensure that the --audit-log-maxbackup argument is set to 10 or as appropriate \(Automated\)) +/Dest [ 601 0 R /XYZ 84.259843 727.220079 0 ] +/Count 0 +/Prev 963 0 R +/Parent 940 0 R +/Next 965 0 R +>> +endobj +965 0 obj +<< +/Title (1.2.25 Ensure that the --audit-log-maxsize argument is set to 100 or as appropriate \(Automated\)) +/Dest [ 604 0 R /XYZ 84.259843 295.970079 0 ] +/Count 0 +/Prev 964 0 R +/Parent 940 0 R +/Next 966 0 R +>> +endobj +966 0 obj +<< +/Title (1.2.26 Ensure that the --request-timeout argument is set as appropriate \(Automated\)) +/Dest [ 609 0 R /XYZ 84.259843 595.970079 0 ] +/Count 0 +/Prev 965 0 R +/Parent 940 0 R +/Next 967 0 R +>> +endobj +967 0 obj +<< +/Title (1.2.27 Ensure that the --service-account-lookup argument is set to true \(Automated\)) +/Dest [ 612 0 R /XYZ 84.259843 164.720079 0 ] +/Count 0 +/Prev 966 0 R +/Parent 940 0 R +/Next 968 0 R +>> +endobj +968 0 obj +<< +/Title (1.2.28 Ensure that the --service-account-key-file argument is set as appropriate \(Automated\)) +/Dest [ 617 0 R /XYZ 84.259843 408.470079 0 ] +/Count 0 +/Prev 967 0 R +/Parent 940 0 R +/Next 969 0 R +>> +endobj +969 0 obj +<< +/Title (1.2.29 Ensure that the --etcd-certfile and --etcd-keyfile arguments are set as appropriate \(Automated\)) +/Dest [ 622 0 R /XYZ 84.259843 689.720079 0 ] +/Count 0 +/Prev 968 0 R +/Parent 940 0 R +/Next 970 0 R +>> +endobj +970 0 obj +<< +/Title (1.2.30 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate \(Automated\)) +/Dest [ 625 0 R /XYZ 84.259843 239.720079 0 ] +/Count 0 +/Prev 969 0 R +/Parent 940 0 R +/Next 971 0 R +>> +endobj +971 0 obj +<< +/Title (1.2.31 Ensure that the --client-ca-file argument is set as appropriate \(Automated\)) +/Dest [ 630 0 R /XYZ 84.259843 483.470079 0 ] +/Count 0 +/Prev 970 0 R +/Parent 940 0 R +/Next 972 0 R +>> +endobj +972 0 obj +<< +/Title (1.2.32 Ensure that the --etcd-cafile argument is set as appropriate \(Automated\)) +/Dest [ 635 0 R /XYZ 84.259843 745.970079 0 ] +/Count 0 +/Prev 971 0 R +/Parent 940 0 R +/Next 973 0 R +>> +endobj +973 0 obj +<< +/Title (1.2.33 Ensure that the --encryption-provider-config argument is set as appropriate \(Automated\)) +/Dest [ 638 0 R /XYZ 84.259843 295.970079 0 ] +/Count 0 +/Prev 972 0 R +/Parent 940 0 R +/Next 974 0 R +>> +endobj +974 0 obj +<< +/Title (1.2.34 Ensure that encryption providers are appropriately configured \(Automated\)) +/Dest [ 643 0 R /XYZ 84.259843 577.220079 0 ] +/Count 0 +/Prev 973 0 R +/Parent 940 0 R +/Next 975 0 R +>> +endobj +975 0 obj +<< +/Title (1.2.35 Ensure that the API Server only makes use of Strong Cryptographic Ciphers \(Automated\)) +/Dest [ 660 0 R /XYZ 84.259843 411.552842 0 ] +/Count 0 +/Prev 974 0 R +/Parent 940 0 R +>> +endobj +976 0 obj +<< +/Title (1.3 Controller Manager) +/Dest [ 674 0 R /XYZ 41.740157 784.430079 0 ] +/Count 7 +/Prev 940 0 R +/First 977 0 R +/Last 983 0 R +/Parent 913 0 R +/Next 984 0 R +>> +endobj +977 0 obj +<< +/Title (1.3.1 Ensure that the --terminated-pod-gc-threshold argument is set as appropriate \(Automated\)) +/Dest [ 674 0 R /XYZ 84.259843 720.651496 0 ] +/Count 0 +/Parent 976 0 R +/Next 978 0 R +>> +endobj +978 0 obj +<< +/Title (1.3.2 Ensure that the --profiling argument is set to false \(Automated\)) +/Dest [ 677 0 R /XYZ 84.259843 727.220079 0 ] +/Count 0 +/Prev 977 0 R +/Parent 976 0 R +/Next 979 0 R +>> +endobj +979 0 obj +<< +/Title (1.3.3 Ensure that the --use-service-account-credentials argument is set to true \(Automated\)) +/Dest [ 680 0 R /XYZ 84.259843 745.970079 0 ] +/Count 0 +/Prev 978 0 R +/Parent 976 0 R +/Next 980 0 R +>> +endobj +980 0 obj +<< +/Title (1.3.4 Ensure that the --service-account-private-key-file argument is set as appropriate \(Automated\)) +/Dest [ 683 0 R /XYZ 84.259843 745.970079 0 ] +/Count 0 +/Prev 979 0 R +/Parent 976 0 R +/Next 981 0 R +>> +endobj +981 0 obj +<< +/Title (1.3.5 Ensure that the --root-ca-file argument is set as appropriate \(Automated\)) +/Dest [ 686 0 R /XYZ 84.259843 727.220079 0 ] +/Count 0 +/Prev 980 0 R +/Parent 976 0 R +/Next 982 0 R +>> +endobj +982 0 obj +<< +/Title (1.3.6 Ensure that the RotateKubeletServerCertificate argument is set to true \(Automated\)) +/Dest [ 689 0 R /XYZ 84.259843 727.220079 0 ] +/Count 0 +/Prev 981 0 R +/Parent 976 0 R +/Next 983 0 R +>> +endobj +983 0 obj +<< +/Title (1.3.7 Ensure that the --bind-address argument is set to 127.0.0.1 \(Automated\)) +/Dest [ 689 0 R /XYZ 84.259843 524.420079 0 ] +/Count 0 +/Prev 982 0 R +/Parent 976 0 R +>> +endobj +984 0 obj +<< +/Title (1.4 Scheduler) +/Dest [ 694 0 R /XYZ 41.740157 784.430079 0 ] +/Count 2 +/Prev 976 0 R +/First 985 0 R +/Last 986 0 R +/Parent 913 0 R +/Next 987 0 R +>> +endobj +985 0 obj +<< +/Title (1.4.1 Ensure that the --profiling argument is set to false \(Automated\)) +/Dest [ 694 0 R /XYZ 84.259843 720.651496 0 ] +/Count 0 +/Parent 984 0 R +/Next 986 0 R +>> +endobj +986 0 obj +<< +/Title (1.4.2 Ensure that the --bind-address argument is set to 127.0.0.1 \(Automated\)) +/Dest [ 694 0 R /XYZ 84.259843 257.404475 0 ] +/Count 0 +/Prev 985 0 R +/Parent 984 0 R +>> +endobj +987 0 obj +<< +/Title (2 Etcd Node Configuration Files) +/Dest [ 700 0 R /XYZ 41.740157 784.430079 0 ] +/Count 7 +/Prev 984 0 R +/First 988 0 R +/Last 994 0 R +/Parent 913 0 R +/Next 995 0 R +>> +endobj +988 0 obj +<< +/Title (2.1 Ensure that the --cert-file and --key-file arguments are set as appropriate \(Automated\)) +/Dest [ 700 0 R /XYZ 84.259843 684.651496 0 ] +/Count 0 +/Parent 987 0 R +/Next 989 0 R +>> +endobj +989 0 obj +<< +/Title (2.2 Ensure that the --client-cert-auth argument is set to true \(Automated\)) +/Dest [ 705 0 R /XYZ 84.259843 464.720079 0 ] +/Count 0 +/Prev 988 0 R +/Parent 987 0 R +/Next 990 0 R +>> +endobj +990 0 obj +<< +/Title (2.3 Ensure that the --auto-tls argument is not set to true \(Automated\)) +/Dest [ 710 0 R /XYZ 84.259843 258.470079 0 ] +/Count 0 +/Prev 989 0 R +/Parent 987 0 R +/Next 991 0 R +>> +endobj +991 0 obj +<< +/Title (2.4 Ensure that the --peer-cert-file and --peer-key-file arguments are set as appropriate \(Automated\)) +/Dest [ 717 0 R /XYZ 84.259843 799.370079 0 ] +/Count 0 +/Prev 990 0 R +/Parent 987 0 R +/Next 992 0 R +>> +endobj +992 0 obj +<< +/Title (2.5 Ensure that the --peer-client-cert-auth argument is set to true \(Automated\)) +/Dest [ 722 0 R /XYZ 84.259843 558.470079 0 ] +/Count 0 +/Prev 991 0 R +/Parent 987 0 R +/Next 993 0 R +>> +endobj +993 0 obj +<< +/Title (2.6 Ensure that the --peer-auto-tls argument is not set to true \(Automated\)) +/Dest [ 727 0 R /XYZ 84.259843 333.470079 0 ] +/Count 0 +/Prev 992 0 R +/Parent 987 0 R +/Next 994 0 R +>> +endobj +994 0 obj +<< +/Title (2.7 Ensure that a unique Certificate Authority is used for etcd \(Automated\)) +/Dest [ 734 0 R /XYZ 84.259843 799.370079 0 ] +/Count 0 +/Prev 993 0 R +/Parent 987 0 R +>> +endobj +995 0 obj +<< +/Title (3.1 Authentication and Authorization) +/Dest [ 741 0 R /XYZ 41.740157 784.430079 0 ] +/Count 1 +/Prev 987 0 R +/First 996 0 R +/Last 996 0 R +/Parent 913 0 R +/Next 997 0 R +>> +endobj +996 0 obj +<< +/Title (3.1.1 Client certificate authentication should not be used for users \(Manual\)) +/Dest [ 741 0 R /XYZ 84.259843 684.651496 0 ] +/Count 0 +/Parent 995 0 R +>> +endobj +997 0 obj +<< +/Title (3.2 Logging) +/Dest [ 743 0 R /XYZ 41.740157 784.430079 0 ] +/Count 2 +/Prev 995 0 R +/First 998 0 R +/Last 999 0 R +/Parent 913 0 R +/Next 1000 0 R +>> +endobj +998 0 obj +<< +/Title (3.2.1 Ensure that a minimal audit policy is created \(Automated\)) +/Dest [ 743 0 R /XYZ 84.259843 720.651496 0 ] +/Count 0 +/Parent 997 0 R +/Next 999 0 R +>> +endobj +999 0 obj +<< +/Title (3.2.2 Ensure that the audit policy covers key security concerns \(Manual\)) +/Dest [ 746 0 R /XYZ 84.259843 333.470079 0 ] +/Count 0 +/Prev 998 0 R +/Parent 997 0 R +>> +endobj +1000 0 obj +<< +/Title (4.1 Worker Node Configuration Files) +/Dest [ 748 0 R /XYZ 41.740157 784.430079 0 ] +/Count 10 +/Prev 997 0 R +/First 1001 0 R +/Last 1010 0 R +/Parent 913 0 R +/Next 1011 0 R +>> +endobj +1001 0 obj +<< +/Title (4.1.1 Ensure that the kubelet service file permissions are set to 644 or more restrictive \(Automated\)) +/Dest [ 748 0 R /XYZ 84.259843 684.651496 0 ] +/Count 0 +/Parent 1000 0 R +/Next 1002 0 R +>> +endobj +1002 0 obj +<< +/Title (4.1.2 Ensure that the kubelet service file ownership is set to root:root \(Automated\)) +/Dest [ 748 0 R /XYZ 84.259843 553.851496 0 ] +/Count 0 +/Prev 1001 0 R +/Parent 1000 0 R +/Next 1003 0 R +>> +endobj +1003 0 obj +<< +/Title (4.1.3 If proxy kubeconfig file exists ensure permissions are set to 644 or more restrictive \(Automated\)) +/Dest [ 748 0 R /XYZ 84.259843 423.051496 0 ] +/Count 0 +/Prev 1002 0 R +/Parent 1000 0 R +/Next 1004 0 R +>> +endobj +1004 0 obj +<< +/Title (4.1.4 Ensure that the proxy kubeconfig file ownership is set to root:root \(Automated\)) +/Dest [ 751 0 R /XYZ 84.259843 763.220079 0 ] +/Count 0 +/Prev 1003 0 R +/Parent 1000 0 R +/Next 1005 0 R +>> +endobj +1005 0 obj +<< +/Title (4.1.5 Ensure that the --kubeconfig kubelet.conf file permissions are set to 644 or more restrictive \(Automated\)) +/Dest [ 751 0 R /XYZ 84.259843 454.079014 0 ] +/Count 0 +/Prev 1004 0 R +/Parent 1000 0 R +/Next 1006 0 R +>> +endobj +1006 0 obj +<< +/Title (4.1.6 Ensure that the --kubeconfig kubelet.conf file ownership is set to root:root \(Automated\)) +/Dest [ 751 0 R /XYZ 84.259843 163.687950 0 ] +/Count 0 +/Prev 1005 0 R +/Parent 1000 0 R +/Next 1007 0 R +>> +endobj +1007 0 obj +<< +/Title (4.1.7 Ensure that the certificate authorities file permissions are set to 644 or more restrictive \(Automated\)) +/Dest [ 755 0 R /XYZ 84.259843 519.629014 0 ] +/Count 0 +/Prev 1006 0 R +/Parent 1000 0 R +/Next 1008 0 R +>> +endobj +1008 0 obj +<< +/Title (4.1.8 Ensure that the client certificate authorities file ownership is set to root:root \(Automated\)) +/Dest [ 764 0 R /XYZ 84.259843 799.370079 0 ] +/Count 0 +/Prev 1007 0 R +/Parent 1000 0 R +/Next 1009 0 R +>> +endobj +1009 0 obj +<< +/Title (4.1.9 Ensure that the kubelet --config configuration file has permissions set to 644 or more restrictive \(Automated\)) +/Dest [ 764 0 R /XYZ 84.259843 383.493907 0 ] +/Count 0 +/Prev 1008 0 R +/Parent 1000 0 R +/Next 1010 0 R +>> +endobj +1010 0 obj +<< +/Title (4.1.10 Ensure that the kubelet --config configuration file ownership is set to root:root \(Automated\)) +/Dest [ 764 0 R /XYZ 84.259843 195.693907 0 ] +/Count 0 +/Prev 1009 0 R +/Parent 1000 0 R +>> +endobj +1011 0 obj +<< +/Title (4.2 Kubelet) +/Dest [ 774 0 R /XYZ 41.740157 784.430079 0 ] +/Count 13 +/Prev 1000 0 R +/First 1012 0 R +/Last 1024 0 R +/Parent 913 0 R +/Next 1025 0 R +>> +endobj +1012 0 obj +<< +/Title (4.2.1 Ensure that the anonymous-auth argument is set to false \(Automated\)) +/Dest [ 774 0 R /XYZ 84.259843 720.651496 0 ] +/Count 0 +/Parent 1011 0 R +/Next 1013 0 R +>> +endobj +1013 0 obj +<< +/Title (4.2.2 Ensure that the --authorization-mode argument is not set to AlwaysAllow \(Automated\)) +/Dest [ 774 0 R /XYZ 84.259843 332.207453 0 ] +/Count 0 +/Prev 1012 0 R +/Parent 1011 0 R +/Next 1014 0 R +>> +endobj +1014 0 obj +<< +/Title (4.2.3 Ensure that the --client-ca-file argument is set as appropriate \(Automated\)) +/Dest [ 779 0 R /XYZ 84.259843 675.273057 0 ] +/Count 0 +/Prev 1013 0 R +/Parent 1011 0 R +/Next 1015 0 R +>> +endobj +1015 0 obj +<< +/Title (4.2.4 Ensure that the --read-only-port argument is set to 0 \(Automated\)) +/Dest [ 779 0 R /XYZ 84.259843 301.829014 0 ] +/Count 0 +/Prev 1014 0 R +/Parent 1011 0 R +/Next 1016 0 R +>> +endobj +1016 0 obj +<< +/Title (4.2.5 Ensure that the --streaming-connection-idle-timeout argument is not set to 0 \(Automated\)) +/Dest [ 784 0 R /XYZ 84.259843 641.326036 0 ] +/Count 0 +/Prev 1015 0 R +/Parent 1011 0 R +/Next 1017 0 R +>> +endobj +1017 0 obj +<< +/Title (4.2.6 Ensure that the --protect-kernel-defaults argument is set to true \(Automated\)) +/Dest [ 790 0 R /XYZ 84.259843 445.970079 0 ] +/Count 0 +/Prev 1016 0 R +/Parent 1011 0 R +/Next 1018 0 R +>> +endobj +1018 0 obj +<< +/Title (4.2.7 Ensure that the --make-iptables-util-chains argument is set to true \(Automated\)) +/Dest [ 794 0 R /XYZ 84.259843 799.370079 0 ] +/Count 0 +/Prev 1017 0 R +/Parent 1011 0 R +/Next 1019 0 R +>> +endobj +1019 0 obj +<< +/Title (4.2.8 Ensure that the --hostname-override argument is not set \(Manual\)) +/Dest [ 794 0 R /XYZ 84.259843 425.926036 0 ] +/Count 0 +/Prev 1018 0 R +/Parent 1011 0 R +/Next 1020 0 R +>> +endobj +1020 0 obj +<< +/Title (4.2.9 Ensure that the --event-qps argument is set to 0 or a level which ensures appropriate event capture \(Automated\)) +/Dest [ 794 0 R /XYZ 84.259843 208.126036 0 ] +/Count 0 +/Prev 1019 0 R +/Parent 1011 0 R +/Next 1021 0 R +>> +endobj +1021 0 obj +<< +/Title (4.2.10 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate \(Automated\)) +/Dest [ 798 0 R /XYZ 84.259843 542.326036 0 ] +/Count 0 +/Prev 1020 0 R +/Parent 1011 0 R +/Next 1022 0 R +>> +endobj +1022 0 obj +<< +/Title (4.2.11 Ensure that the --rotate-certificates argument is not set to false \(Automated\)) +/Dest [ 798 0 R /XYZ 84.259843 138.881993 0 ] +/Count 0 +/Prev 1021 0 R +/Parent 1011 0 R +/Next 1023 0 R +>> +endobj +1023 0 obj +<< +/Title (4.2.12 Verify that the RotateKubeletServerCertificate argument is set to true \(Automated\)) +/Dest [ 808 0 R /XYZ 84.259843 670.970079 0 ] +/Count 0 +/Prev 1022 0 R +/Parent 1011 0 R +/Next 1024 0 R +>> +endobj +1024 0 obj +<< +/Title (4.2.13 Ensure that the Kubelet only makes use of Strong Cryptographic Ciphers \(Automated\)) +/Dest [ 808 0 R /XYZ 84.259843 392.223057 0 ] +/Count 0 +/Prev 1023 0 R +/Parent 1011 0 R +>> +endobj +1025 0 obj +<< +/Title (5.1 RBAC and Service Accounts) +/Dest [ 815 0 R /XYZ 41.740157 784.430079 0 ] +/Count 6 +/Prev 1011 0 R +/First 1026 0 R +/Last 1031 0 R +/Parent 913 0 R +/Next 1032 0 R +>> +endobj +1026 0 obj +<< +/Title (5.1.1 Ensure that the cluster-admin role is only used where required \(Manual\)) +/Dest [ 815 0 R /XYZ 84.259843 684.651496 0 ] +/Count 0 +/Parent 1025 0 R +/Next 1027 0 R +>> +endobj +1027 0 obj +<< +/Title (5.1.2 Minimize access to secrets \(Manual\)) +/Dest [ 815 0 R /XYZ 84.259843 523.851496 0 ] +/Count 0 +/Prev 1026 0 R +/Parent 1025 0 R +/Next 1028 0 R +>> +endobj +1028 0 obj +<< +/Title (5.1.3 Minimize wildcard use in Roles and ClusterRoles \(Manual\)) +/Dest [ 815 0 R /XYZ 84.259843 423.051496 0 ] +/Count 0 +/Prev 1027 0 R +/Parent 1025 0 R +/Next 1029 0 R +>> +endobj +1029 0 obj +<< +/Title (5.1.4 Minimize access to create pods \(Manual\)) +/Dest [ 815 0 R /XYZ 84.259843 307.251496 0 ] +/Count 0 +/Prev 1028 0 R +/Parent 1025 0 R +/Next 1030 0 R +>> +endobj +1030 0 obj +<< +/Title (5.1.5 Ensure that default service accounts are not actively used. \(Automated\)) +/Dest [ 815 0 R /XYZ 84.259843 206.451496 0 ] +/Count 0 +/Prev 1029 0 R +/Parent 1025 0 R +/Next 1031 0 R +>> +endobj +1031 0 obj +<< +/Title (5.1.6 Ensure that Service Account Tokens are only mounted where necessary \(Manual\)) +/Dest [ 842 0 R /XYZ 84.259843 449.446885 0 ] +/Count 0 +/Prev 1030 0 R +/Parent 1025 0 R +>> +endobj +1032 0 obj +<< +/Title (5.2 Pod Security Policies) +/Dest [ 853 0 R /XYZ 41.740157 784.430079 0 ] +/Count 9 +/Prev 1025 0 R +/First 1033 0 R +/Last 1041 0 R +/Parent 913 0 R +/Next 1042 0 R +>> +endobj +1033 0 obj +<< +/Title (5.2.1 Minimize the admission of privileged containers \(Manual\)) +/Dest [ 853 0 R /XYZ 84.259843 720.651496 0 ] +/Count 0 +/Parent 1032 0 R +/Next 1034 0 R +>> +endobj +1034 0 obj +<< +/Title (5.2.2 Minimize the admission of containers wishing to share the host process ID namespace \(Automated\)) +/Dest [ 853 0 R /XYZ 84.259843 589.851496 0 ] +/Count 0 +/Prev 1033 0 R +/Parent 1032 0 R +/Next 1035 0 R +>> +endobj +1035 0 obj +<< +/Title (5.2.3 Minimize the admission of containers wishing to share the host IPC namespace \(Automated\)) +/Dest [ 853 0 R /XYZ 84.259843 238.710432 0 ] +/Count 0 +/Prev 1034 0 R +/Parent 1032 0 R +/Next 1036 0 R +>> +endobj +1036 0 obj +<< +/Title (5.2.4 Minimize the admission of containers wishing to share the host network namespace \(Automated\)) +/Dest [ 856 0 R /XYZ 84.259843 603.629014 0 ] +/Count 0 +/Prev 1035 0 R +/Parent 1032 0 R +/Next 1037 0 R +>> +endobj +1037 0 obj +<< +/Title (5.2.5 Minimize the admission of containers with allowPrivilegeEscalation \(Automated\)) +/Dest [ 856 0 R /XYZ 84.259843 252.487950 0 ] +/Count 0 +/Prev 1036 0 R +/Parent 1032 0 R +/Next 1038 0 R +>> +endobj +1038 0 obj +<< +/Title (5.2.6 Minimize the admission of root containers \(Manual\)) +/Dest [ 860 0 R /XYZ 84.259843 584.681993 0 ] +/Count 0 +/Prev 1037 0 R +/Parent 1032 0 R +/Next 1039 0 R +>> +endobj +1039 0 obj +<< +/Title (5.2.7 Minimize the admission of containers with the NET_RAW capability \(Manual\)) +/Dest [ 860 0 R /XYZ 84.259843 453.881993 0 ] +/Count 0 +/Prev 1038 0 R +/Parent 1032 0 R +/Next 1040 0 R +>> +endobj +1040 0 obj +<< +/Title (5.2.8 Minimize the admission of containers with added capabilities \(Manual\)) +/Dest [ 860 0 R /XYZ 84.259843 323.081993 0 ] +/Count 0 +/Prev 1039 0 R +/Parent 1032 0 R +/Next 1041 0 R +>> +endobj +1041 0 obj +<< +/Title (5.2.9 Minimize the admission of containers with capabilities assigned \(Manual\)) +/Dest [ 860 0 R /XYZ 84.259843 207.281993 0 ] +/Count 0 +/Prev 1040 0 R +/Parent 1032 0 R +>> +endobj +1042 0 obj +<< +/Title (5.3 Network Policies and CNI) +/Dest [ 863 0 R /XYZ 41.740157 784.430079 0 ] +/Count 2 +/Prev 1032 0 R +/First 1043 0 R +/Last 1044 0 R +/Parent 913 0 R +/Next 1045 0 R +>> +endobj +1043 0 obj +<< +/Title (5.3.1 Ensure that the CNI in use supports Network Policies \(Manual\)) +/Dest [ 863 0 R /XYZ 84.259843 684.651496 0 ] +/Count 0 +/Parent 1042 0 R +/Next 1044 0 R +>> +endobj +1044 0 obj +<< +/Title (5.3.2 Ensure that all Namespaces have Network Policies defined \(Automated\)) +/Dest [ 863 0 R /XYZ 84.259843 538.851496 0 ] +/Count 0 +/Prev 1043 0 R +/Parent 1042 0 R +>> +endobj +1045 0 obj +<< +/Title (5.4 Secrets Management) +/Dest [ 887 0 R /XYZ 41.740157 784.430079 0 ] +/Count 2 +/Prev 1042 0 R +/First 1046 0 R +/Last 1047 0 R +/Parent 913 0 R +/Next 1048 0 R +>> +endobj +1046 0 obj +<< +/Title (5.4.1 Prefer using secrets as files over secrets as environment variables \(Manual\)) +/Dest [ 887 0 R /XYZ 84.259843 684.651496 0 ] +/Count 0 +/Parent 1045 0 R +/Next 1047 0 R +>> +endobj +1047 0 obj +<< +/Title (5.4.2 Consider external secret storage \(Manual\)) +/Dest [ 887 0 R /XYZ 84.259843 568.851496 0 ] +/Count 0 +/Prev 1046 0 R +/Parent 1045 0 R +>> +endobj +1048 0 obj +<< +/Title (5.5 Extensible Admission Control) +/Dest [ 889 0 R /XYZ 41.740157 784.430079 0 ] +/Count 1 +/Prev 1045 0 R +/First 1049 0 R +/Last 1049 0 R +/Parent 913 0 R +/Next 1050 0 R +>> +endobj +1049 0 obj +<< +/Title (5.5.1 Configure Image Provenance using ImagePolicyWebhook admission controller \(Manual\)) +/Dest [ 889 0 R /XYZ 84.259843 684.651496 0 ] +/Count 0 +/Parent 1048 0 R +>> +endobj +1050 0 obj +<< +/Title (5.7 General Policies) +/Dest [ 891 0 R /XYZ 41.740157 784.430079 0 ] +/Count 4 +/Prev 1048 0 R +/First 1051 0 R +/Last 1054 0 R +/Parent 913 0 R +>> +endobj +1051 0 obj +<< +/Title (5.7.1 Create administrative boundaries between resources using namespaces \(Manual\)) +/Dest [ 891 0 R /XYZ 84.259843 720.651496 0 ] +/Count 0 +/Parent 1050 0 R +/Next 1052 0 R +>> +endobj +1052 0 obj +<< +/Title (5.7.2 Ensure that the seccomp profile is set to docker/default in your pod definitions \(Manual\)) +/Dest [ 891 0 R /XYZ 84.259843 604.851496 0 ] +/Count 0 +/Prev 1051 0 R +/Parent 1050 0 R +/Next 1053 0 R +>> +endobj +1053 0 obj +<< +/Title (5.7.3 Apply Security Context to Your Pods and Containers \(Manual\)) +/Dest [ 891 0 R /XYZ 84.259843 324.051496 0 ] +/Count 0 +/Prev 1052 0 R +/Parent 1050 0 R +/Next 1054 0 R +>> +endobj +1054 0 obj +<< +/Title (5.7.4 The default namespace should not be used \(Automated\)) +/Dest [ 891 0 R /XYZ 84.259843 193.251496 0 ] +/Count 0 +/Prev 1053 0 R +/Parent 1050 0 R +>> +endobj +1055 0 obj +<< +/Count 142 +/First 913 0 R +/Last 913 0 R +>> +endobj +1056 0 obj +<< +/Length1 41004 +/Filter /FlateDecode +/Length 7589 +>> +stream +x| xUut;YH',-11 EB$!`B'AIT&d 0Ȍ +88L@f:cLnx9ֽrι{?R R23[1i& ǩ=x~sg7;@>x>|- `wbkͷSF'N+KX-NSE@7T^7y\L`ySgξ}/{6eꌅy3 *ϟ}7*zSTKsr> @pFEiY3s+ ,\R{ϬSy `=m&s3|Pu'<\G PɕMƎKIEfJ H[&*DC +4i#w# `QD|77(|;FLv)XVx[ +_p rߧe)s*`i +:iO(N(svx{̦1f0[xqpD'[yZ#j5nQ`,gz‹:tСC:tСC:tСC:tСC:tСC:tСC:tСC:tСC:tСC:to%2Ckmԯ9)e]twj, + Ca4@Lߪ~=6zzt= 0洶^9~7ADtj*r<|ht(^*W:4׊|oxUx~5_/_=^io.xqwӻp'Ǘ8݁8*[%Iun |0\|nMz. 7p 帾*@Zϱ*+Se{*Kgt8*XQ>Q8KxXŧ8YSZquO||n,[eʜʌTQ+IS+Ž|]Zqดc G_KHs\bTEr|̊ ,8߄8m¢&ӄcYgDtHiT*q1d8&|؂9ǙM8΄IX 1.lPIs2}GkFr׆#8'9jЈi `U8 +rG!ӄDZ?ǻtHw9Ό@Nf 2|/rN&&vbtˆf=0663$L6cF {{*vꕺb$ՋILLJwcf)>f1&Ihzr1 #I\@7i1 0*q T`.NANm 梕c%X +h֖`4s4Qf +G\HBEN^& =8Z)/"X:g;~u6T `̣7V&p QR8E` j>As>YqxP%֊ų[,ϊbKíXi]}M! +jE(#-a8PE/~I2,$߲u_YCWr ?<6:Fխ3'qӒK*HgdL!0FV6(CtI'tI'tI'tI'tI'tI'tI'tI'tI't鿟L4A.(~DwoƔhWt|SJW Px$7Ns~zKqtL`(TKOڨDn(f^8f\514ǰ( +{Gۥto8lÆ5o- ta7 PE 6ºc.`$qm}]@sL3Cl(Gؚ׸ 51"NOo<~uMb҇x> a0.|y^e2lYe;ag}"L齠w^Ȇ^RES6ʄ#Si9s-:?0~} ھ3]P5woKwm/[/U$`/.HmՖ5^Kp/-I@kpvU%:<gY,-@0Md12\ļ*rB6N ڼy]8o w`L׾-*b#~Gn9zt#ڸx7݃w3{-dHoUlR>Y@6;`'DA]Sf7  вZiIQ 4T2|(}Q2 Q5|# zcQ;n?kX@C`>0"hY0,SY/,9cJ)[%,<+l*1Vhb1cz~U⯅>jydd͍$v=04+OJQv"EI`uKaFp;hZf:d}4ӥ]r 0d`hmb;2^Z-)c3l/"g֭]rŕZN|^z +!U0gv˶:=g}}E0źTig[`X־!,R.*ip5``xT@fCzdàZ_c6.=zedW^&]xa~?~ܐ!l3u`q>>H+>Wu&Rv1-4l$ޚOvp_GGe13Tͷyl5{if((l^ͯ+,9h)fYlÐ?>{KZk 5B g4ϨV H|R-K%ybP=;HƑ8A$QD.8@ x,J2n),Lw1B F:cG: ]<6Ǣ &$.1DyxL Jً2Y/I Vɫ X{=ڣ&M,>uw]g?~}>{͠yyKi>qJ&I>#}-WK/_U$㾌`5 QQ0&sdb\Zi+a 61d +7@Lx!hdcscGM/ ¯)'0s(*)9$۷?CB MoKK WX--nr}a!ODf^KOCޠUZU[(&{7POa9h]!rYu][k_b 0*(Ti=ߘjזdfoz2JHLJDOE"}Q>/ɊΊJ(LXQYU)^SPp-!k{ّQٞ¨BOaȥQK=KCo:s{c/ĐzEw>QgZ~bKrl¿_'}|0iDrNl}duϞUz / /VUue*S] U#8jhw^6~)>s`-|ixExu:-rZisbxy Y֫Y/w!{utnvű9XXZAJ ; t.u`uIk-[O aadh~vjʡȥՑ:6 vobݿLk[N;w>KxHxƝ9l Spי˗m|8ƕ*ɵIuv4QkYw K#p[cOFRˆ"EH?/wdcwSUn8[Uptpn5LчASA,i>`9{S&7_XRWP+l_*<<ةo׵;3??M$` ]x[1Kf*כ hTݲIuf-jy5d8ٜqC8 t ۮR!\.[lzj?j&˒u55^'7'#!Pu/ V&Z&&(0RݹNg_WF2сlvڵvQ&ͮ1vʽuu^w-a[C¦͛mk)-~ RBYatYc~$ؚ3Uskhn$},cHHvz8QKBx&EI]%]%GsI@FYbLffU')eNFh6-B{DLJ $IݔB0U)H˄273B^yQpP<( f1AL >K>f[pX&=%6zn|I:dxx .^L x1b ޿_k9U';K-[ +-7s 4oH[!^(]dHa6]"h_0apYJ#-bv.IDMV[ϭjfn[b4j~45C u:ք5(TsDmwlb(K %JTb.XKKl%bGuص0o= {uh=5WxӗyԷy#Bny'ïFځ^d3KzIV@t(`V U'($E Kuu}ƃc-Hy%f+s͖bmwes,T=Ｕ7bQՠ8c%\){t>nən$7b DsנdgkpbdbT'):.aiyeeCUBdlF цa-F'=T4":ZR(o ` KsBY٤uOnGޘVβ5S<%&mC?<WlH{Ng.z?WJMf16Ur`H% ^g4 "IQz*VW /z [KKMw-e#7n@.gwy8m9}c1zg~jHf6 3oQB<ۧnkk?숰%gcۄ⟪M;U=^"f ?8$ $ANڏ.![:g(o-"<* %JaV&(DFzW9`1Y&%Y_+(ɆP1S"0bylNJr2b,E +Xʰ΃rTF|V|VZ/^kEUkz 077CMbvS?vC|AZa ThsWT6lj@>1!t)rTq5‰ԔBNaj>Zvܠj&'ijo/c XCP.G:EE`< ~yPC!ZC﷮7BTnrwT + Z)(ئe]WTS#v2|Iɢ(6agO厢]FB8>@PWLSݭ0L:~F,RMј!fOt6>S]JFHcL7u2UM5uGSTX6XlVTڜT_ݫw׼%z>N^ဪζG&-p%lWnhBn5a/813@:ܲnI2M0 Bݞ]vez^nV-N;E2.kC!LPh#I7jA0yٖNbld[^'˃{ +a!́i0a.x & x R(7Zx` Et́)3;YԾ'D1VVB)g> f1i>ͥ,jC}RQ7Q >9hS9Def;y͞=u(EtUg-Ҿ(4K'M[?Uk}c҂r~eSi]}baꗏl1P= #FiY>Ӿ)hݺSNZf@J3i $ 2x [UDd00sș; BA$PԞS[S"\3z&hwFQ[ +a qڦsLigЇ/5pGU7?D˭[_ƶ9-֩ŶI TNE~Fsm>ܩ-zPFKZRCj:utkο0<#+t'#~! Q;/@i{Lo{kyDzHd5L#=@;x"U}A(R2-Ue!meX:Ϧj5eߖƝ qv^;qSN97?gHFGDDM)> +endstream +endobj +1057 0 obj +<< +/Length1 7648 +/Filter /FlateDecode +/Length 4712 +>> +stream +x9yxSU眛&-PhZHYӄ(rtO&͞nItInPR"ouFD}*OE2 ~CGE8qDܼ߹IK|{[o;)h1Ho.)vW;V!4^]dv,(; S])jnBn_/.A(l5+=*A9{%ˏewr|i a=J y%NFV%0f%k#B9d5mQd~kFNF)$9wПȉJ*`y1;$Q<9{ 2g_gA10;x^~cԎ1l"T'D҅IQ}H -!5*h܅`B Ff?HB&ZXp*u61 +7"4E 3)2\ +A5h ZhN PhV VUl]p6˧~k~KڎGoA9>CWCI&ny/ALa1_ R킇''87|_وyDcƈ"x7iis,[T?R@#fM$WP*3Mx`.'8PiS"\dccdQk5:r, B$ZQZc ]e(ՖԉLn[VZInN,Aiixs2]U¡/\:<3ܑ%ja,WXF(9_;#/ 3]%sB(HNg%B"-vuyq#8Νw#V`eupT֨RFKfQjkg_o]{_glq9}RKm0#IEJ<]LWKAxb] QYrɣY2fevB8Гhc͵pǑ7 [pv|E-\Ej66UJf % +E|Pj+>)`_㾪bkԧڧEoQ$$M r@7IBUXhVgm16|Jd65KouQnKsktՑaQLzeKÚe/plFׂ #4"49XJ -uMMz +%NQo>m4[!$He87KtnZjKҮ,k4*ҭ/.))GȖ vZ 3uEEr h8t ptBĀ&A%SLJ5PUk5dz9R*L'ynuhK?Wv> 7 +[QJkm}Rmidg/F. \x%  rh'{Bp2 +*B Rh;tVgկr8Ub*,q\zjVl6BSgi4|C} caƩB}Q` jУ8ZH9A )'TU*&C{κ:kZgV&WYѝM'ylEO.[iVR-x;Sr(iFI7AA nw$*r::zAd#y!X&ǰ1r%?qg2꯷_/y~s{t8>G[&qc6[m5qׯTt{Ho2ŧx{nz!۳^A1\̆(xסC=;G;;wȘ7?O)]p# G,8<`\(46|{o=dyQHfZ 5{U*k˖ܟPw׷foJulaMʹb xo9i +NZ#ЎTޤxp#pj9dVVðFO> h Rɋ<T=[[{#.z^F!=[K&GULNE"=jFl_Z~u+9}L7n u2PC-.L)tIQ(ʎJmg5qR+wn>sܦO{F$] +Τ7V"GxYqz۰fڷdLͶDsq5Wg G /Ǥ3IVM,lb|BWvVa}xS-7ӒY"JL3KTqԆ%ٚ9Q1luPJq8!GWOz^GVڨ؁J D|SZlBasMuu ,CYYC!v7_lL GA3"|@RfD 3Ovn&t()#NX\c(fAh>\h>r+⾇Kӳ죘UĈFUq,>Q\l!rLaC2 ggگ;t„ibaqq:`~m큭d}CeMg@%f&9$Jͥ}~_!$ #~ds2hZ:޹fj#>*gYJfrn`ҥ4ϷY6ri^]^MZZ MVp &'Ni&\wVs.A@E(u n&.\>(`BXkM+܈\͒9EE9%b4 )+ ΆƮ 4bBZHKҲ{惴:֡[H+;給@\A\mo"Ԑ; M5e$la͖(`hɸBWFBlWVQ{q-' &h,͚$ +x|SsvܑU]ug!J1 =~)l)xedMbz6(e;z|1yV( ]OC薶C֏_Ԫ H"w_)2*hMry\$. 'a5:(CY5Feh 0/οs_s\\σ~0$dOCo̢B<qҎIo81YG'~n\56?I"^TAl כ#EE#VhQ +ڝr"@ G7Jޤ$o{ :x[y Λ=ЩΐN:}@y)܃̒)Z[zVO +s7bgKnnO+Ss%JڃDǿȃ{JFc +LSS+(P t7+0y,&%ۦ*OvDST,WQ|2RoIzY\$2jNdY +;yj"Sr!SBVb2Ca6{vyii*m4Vox]*szj2-٠jV7Qϗ.K[k8J`?ǰxg o< ,A>%(ݲHjXG1c(85>ͱtHxmӐ)fǰt=ںi]3GOQx[oqqd1K X,枯DqCNpF".GvJ^qiwn6@6v3HA!^x:;`|~I"^O>t8u[]j oC?M7rз-@WCah--D p<2 ~ԎZQB|]0 m  0=hH5+*h`_4+PDػX 7knc- +m06қe໐@ [8Bs*5АOoNK .x_|oLCpC*S>ԼZk?5G" +endstream +endobj +1058 0 obj +<< +/Length1 47128 +/Filter /FlateDecode +/Length 11768 +>> +stream +x} xEhUw2353IH0<W $0 s1bVA +nD֋.rO$XoΡNwUPBH + 3e㫞$Xҷw_}p?+< B{~0!/oCΌYЭx?|Œ H'dxvmsyF݄affrD|^5xv9 myyOr7%֭X>i[N1"o=lBB!3o/̖c 8xFI[۵Hse4 p||VI_2E)"y^8gIwO/ϟ ٦FjUL2${"lCMkrfC`M`E"߱PeK8t6-zH)0ȶf})=I"U=CXf/mhhOHiSK[I5J 2؈xI<*1:36ڑHsYm^ϰ'W'er^A>Hͻ 54xdC}y AF|i^ Ul2 ^k6Y+v:.$`H,kלA b 1A b 1A b 1A b 1A b 1A b 1A b 1A b 1A b 1A b7@c;KgzKj"F\X AEޤ?La2#S- iEڑk@|> 7iy×^5E` Yȍ#o k{me V!y CF(|d5OVӿ_k;pȈnڪwAc +x){<U'L>z0냮-DaƲ% !8mIj.lXl'2id"24ɬeY&,ԃRkkuY:,NN^Syf' +Q+JOEXg:`ojfi!6o;kϯ9/n +1%4,߀Pw-p- +B_BQ ? QyeA}]Pc@MD۷[|k_kG|_Wpo8-K%  8%Z8i"?9>MaI-|>|q]p+?glZx? GpO~$^ދp*C xB"H[av!Q +ÛޘAkjW^}6+{$u{le'K8Kv x /Sx>xyi{-J6Ouz|2BPv(w);EREg+ke>!)! kְ5i{ t 8-\fзAw[?hrY.=KFWa7U832[^R~ TfЎG PR-~E#v8f&*UWw0'.?#qdP%2'/_E-?r=wɓy]0{Nfun1*nr;̐/a͋h֪_-~8K,VѻB3 X ?m_ddM2Φh:s|V3oow爕$H^*ZD%8O"$r>_:M]-T)'H(Hf ۧ,X0uS]bD|,ͤ4jh3qohX"ft6Cr~_ "^خ|l׵D5 H"VMs3a;th:XC xU]?a QNC,p%S9W<89UqkKr|^Ӂa#^"аոFǻ7߹-.4ZZ'GOLHHHlAZ$H% 'ORI7,yiqu +*^ja#_g}Tߊ;fy&34~΂)DiW<3|K,ΰiCUJy{\x>*:}*ߗAiYc5qm>%,~<:M}>G t%鹍i==!Ğ_:՛ڴf#{}cg}>Ѓ<⳯ae)F:&)1Pu NB0e;D6$bI~&ckмQk Aɝ.~:~\VpT#BHVWx,;9ľ* Z3ĕywu*Li^+L^lʟq it~\fߟAC;@m۴VZh-IH TԪJ +WU)6=$BeXZiك&yRg?"mz=PwIc5;SPZRBouH~W5PxS\'ZA{Oz}qw#DM6Q8ES,TPj&YJzGS,̑J,O'H!3FH }Ƈ4DC(%{,#?EEكn!'bihˬ"jUn*;ѫ۠eu7COJ΄$gFe6z7"Eޡ \ɱpFFn疡F#2s^粀;̙>m,`4 I-^8:ϐ}=Ä1Ə5f[{̙3**U*e~eg_~ٲW#FW$e?@'FeoWd%]T.VVS–"n`]D6svQ$s[d&q:j`ͨ3kO-%Ş)ieI)*Eiizڠsr6JN w{w#gJłQVSxU*]icxNC)  d؋Ac+T[RUUkCKTP +ٮ)KTf蚅+] !;@ |QLDxFEFcQjTJZUsuƫGO8-EZk=h}~ 編aR'E9d>C9}>}/h/7]k{C!r\D` QASeXȲ,K,JW.qmlݽM@iMdH$G^rS_JZ ĥٜzbb_uY|~-AZB4*H)ώ褅ECZ(QgM% qǎ}T̩/ob.֋v |ogP\,1ז?Ȍ7JQǠ-IHͥ1vmv}-Jog#j$fuMi8z#Js- 79|O3!1 :?iiC\C7ZJ7\Eۿ@\kq?j#T7y@Gdig՞tfJ4q.{?1nٓU`sN]T#NUvϾsJą(S,s%C鰒GٯcmA$`3x`_pهM&q⬟&կf/sR|ǶhcfErQ́a]FcbDH䚑>^*.6juwE505:PTMijFq f}ܧlݡe R-EJRi)ݫܧn$dMjH>BR>C+gUj%3Z|9ygW;_o֟bug٠]͹IB*9PPQO7.ٕu{%6H'ȳhG&OtCy !3\׀Qn#A=`#r\.y&N3>̩9ZZ@mXO֏YdVf)3Ve2iVm6`~p9Z0tp@_m6RL'A6U/rܫ=?%YkoQaӿGbXΏ}6JA _CB' "<`yE иJ-bYY7:Y.sPRlW}u8N"$oX'K|r3e)eS,VSTJ${; bh&D!H13*g$tl-doPNjlps!xD:>%.jҎRJz;(GUo>1nĉ= 1=K +?R\$$~K!umIOW_%=[|VH;:I>B^&H $'&Ry޾!i%_Q&9OB|БSLQTA_솅E ]n =GM*!?տˏU%FCQ#go=<iMŠ'mXx_OV׺WJ3d I;<VOk9(#RJ=יK*;2<GcG&˂3Cssls  B+O[X\bbbߖ-m[`[p[Ut""j9qff4)Qw;{FVlw⿖.w7zϏ u̧ +"vfА%rriAgPZ(;N #0SͼSġF5گjtt|MN6/;y ./VvD{63*fqy"L%*Mai_I`~lR 1qBeD#m s f + $DHTx@ j}I_):j&BC>Lw$rёPGGBҗj*R$/5Ү!<\Wy+ +̆qF.VY1~1j!vC}VY.M7=/0^AĮ膻򞼓1q,ˋy13sjDHPܫy$k-Lǔ#Gj%[;{:Cp7+7PPm~VFh ɖm>͘ib+ͅmm-j^d{e7^gxI.fIC= 3zQ .΋Jy-J5VDZ]_4Ih~̢t>բ7.E4EaiTLȨlaCXj z$֐KBqQfH6Ac?`FԿeF[_zGDB ]*u c7+Å8o _*o(ѪHk]%9ሺq<x7e]!*IODn-C.`s0]ulP C3m3Tp'B |%I\ ܔ6$0!(poڢei;Ҫ..csP\~9lco,yPkǸzm}ͳC;P;0h@Ⱥ-n|cw7oO)˿MG>V`nB$lGdwPģ~I$rGS)slҊdlE +&Hp͞=*PCtb~uO[4;>P(_,AJY墬N"1q~V6$,5qɕr=q95BO=rca"0704''x.8Nj% 1rUĹJ@:QmY$i +ϟq+31)i"8"~Ƒ9;7J͚7?l0ؿ=yD+dr]$Da_g%;d[eN +-Ť`ɹr%3U bZqeڇona%ƪ9:!-[Tz4&8UtaL)L\.AƓ#).Nm#N[Ev{omV^u?x0=SW:+FM#|eƐjEo4T:7u9Ob͎?T=R0ԭ"J)Wч;y(Vd`$# {w̿..A?dߙ!r2lxnYMAr:*eY[n\Ϛ[7(Pti{mT{'V]~Iӽ񅙯ö/LJ|vuǪ~³/Tݤ,̈N&)b7)ыr}\΃hCQ.W=y pcKwMM S;q]0=-E~c,e#,̥i%ܦܦη,{CuTI>]֗ІBmF>~?=׼W.Qj?bXzꞧSH?shJڋdd_2r72dFZ6\B)doXHP$:Oá)h:8fWyqTWFt֓*vv[.@І!噋2e>i)ܽ4%ϕ=/8$VQgpi"C'^LLqᬟ=%i?aL{Iٝz>H"Re}h&R!.n#lV%꽜U?U{K|Oxhє%܅ V,XP +[,NK/fÆgAx[ Mk$%Ub88-z9rT134ʮӧ5Kheq'Bf ѳ1lߥd"ctcI_i֘/$;?sJ5 ͢`mRD6Ҹjty@Z8j6aune]MCq͕3>s٣%/kG{)gXO'3} 7qk]|s3V7^L\QW?X,r+:J"5]#Es~N0򁁶OS I -iRHgJև"v~Օd%] |^/A G'bRTW r0nmWGtGt~D_ݏ8rOVW^#h\m57ӍJdG4}㨁  >Fw.&N>km,Foi :8rI.̺jWK[OG_~pm>:6QʊH-i"k+WfrR1ʭ(>CGm[ӰEyA{ZbfC/h/ޤz} )3_FŠ7@1c1b +z]\N4jy>FXTM'dsWTEURꦢI_E$2,j5&V{.0;xi'?'Bt t;Aw:1xφ;`,їꏑe S^ _%pD?pJ??9ڠb*6 xhFBWh- Mld @eV`#Rux`kmZ^U^դ^QhQvJzJ;j|M>g+ߨk?ԮRSh[b::[Žu?{Y.vAIiw~\Qšܜ4/r38lvp 930c .êk#w5 L3 '8LeMb,5ST/;,Te=j:]ֱ͜FP8G pGܫz--yKE5Yޤ7޼:}bbcl%..VW;mMyt/VjUKow:q~ΰK = <]õS*7%]kżWr} !JN$ih c &mTѽq_A##pVjncYnP5EZ˚YZk:@g<kڗP͉xriޯw5 FT'IDIn9=)=;>\0buw$wbJpC37y-Ii*4օ7[nXN@n.yBMw~j tn58(r+UߒS +~EtߠiLmC2t~2W ~]V:u{|o{XRHHOgq )dIF}.yr5&˓#5 oq\Xnxs`W&cbxZuOiKq5˜T{I$D3N+q锯Fz]nЕud[ig&v_LJwa[HAݩ_|mȵ,E -I{2cgʖ,@pt 6~˱2He*67t& /FjmXְ_ܡMô_o0O\xe X~+{b1WԿϣ\Qj~'Wz5ؼi䦸 Ήsչ<@]K\@U\12yHzT]"p*.ɹKslن ݛkPK忐ׯAEiO\.^FZ/m.m-˄Q2Kt@޾[?Kd$z1?ʗ+KSp_Qamjz7Uc:.]9/]hn<Ǒx"?I͈[pd7"CϘDTL‘< )G +"oQih#%!7 :1> žoFt/\Gأ81HCԗ>Iw*OϚ1,?x +Y6xl,'rެd٤24dY6x&\2k,&˦e>s&e3ɚI3ȲY3ҙ,mrIWnD}7E3{H +iAZal yPsc9(Y,ߩDY +endstream +endobj +1059 0 obj +<< +/Length1 17264 +/Filter /FlateDecode +/Length 12304 +>> +stream +x|yxTŶo {s;ݙ:N!d" @0 a<*C &P"ǃA<!=9\q@#z= +݉N]VZހ0B;̔=/ ۻO~{ :dx.FAc8h_ CTnh:oB|qS,j "DR/1o,s ̘h!aCm1w&G ˚ ' >pu9'3Ȅ]0ur65pq'ZNE^~{E¹.X؝X~}v;yF $G .+2 +Ds  I( ME#9y~&1G;3uC&Ȃ$@Y *ftc4D/""7tzbC1hdgvU;lS'%4 J8oPA{W~ J1pn:PnBe]\E|UPVϢu(PPSAZyc(#.޻88&?eS~?H@xLj6:v\+!ԂjTs~ҳ~虧HP>UҦ2iFO-2zw=(62zcQ精1І[(:O +i}E/MFPw}`bD^F_cV*{2nJ:m/ﲊ/']Vחſ./%ӿ0&/Xa }^lctVo,V33*isJFz0}DW ^tpNi9(/e 2/}J1RF׬k]m*FO +-yrq^N]fK}]A\l1]3ǎ +ltOxG]tDqp:/etN:t:qNgtŌN2:Y)n:INdt8Dqұkc4NG1:ч#ćF3:чӡ FiB?>h ]_؏Ѿ[L Upqpw6}0*>hO'2V_OzLb/3UB˗g}&%3<ͫ>_Г@B[49 b5-\:f3l@] 8d073ij8MZvօN!b&>l'UB;qr+bxiGMh" ƛ74_4XYe,zӘd=ad7QFl#Yጆ1b =8 h#Y ͧEt$S}!vjmZsk>+dg6D3Ud ;J AG \=*p2 ʌJеĨh9$@NBq=.^'C K-鈟)%+m,Mx>]Ugғ8j lhd-os>78³KۺGcq/ Gy% \V8a2z¶ @S +迈2tm!Wk +M詖 VH@,6$p]lpyY %Kv9rۅid0:/C#8 'UTAf3bdGr%$m[#Op- +}NBm8F&sdk^-Fp?e@'@2-0L\P-lzZ kN2&Hs퍲a(ZH^>$ {n0KFj3MGEB0Y8(đK)Y~rwyK S&^]TU5M}J}UDD?lVݍLѦѦ3̵5(Emn=da+=ck Y=>ݾcB'|ەZu#lҢsE̴TZ*1j9ےR[RqQk#MMMbs#lZ@Z +[FQ:BAo% YCmtkg_N@xѴ (EH\NQ]yYo5F"#]b6 N pc9{>7ן-0 ɠVBx2=Zq4ycVͬإ5n11qSfbƈ&lVM/Ro;ZOu;'-b_]מbq`oY щn{Y\a-A$""&Ƙ@pnʩg?r y'kkܼfXs6'dVVf$9V||Bfh%+;3 :L/f.wO?5i껧)Cvxš[7L+?,~^X8ad:w<3u6}5)E`Eۺ:~fn= +GnrH_;;*16AuϚcޔlMYWdD̤:H5x.h5xbƮpL^pnո'\ +ՙ +al1/P1W -θ 䆸*i +Ņ m5Yggee8\OK6~h'SOkUA}&/5n)"BZϦUΌ)^wW +zc{+t\7cu_y/NfabA.71Z+]pҨ4%a\~%DjvdIȥ+. jqmθxŖM^juMU +W]-87_+c\M[7#9.˖مqZ5лdka.ܖo;_^._HNƔC>/Z ϫTiS *IV +3 z]Qi&s(N(pq3S(^f WHb[oۗ9x"JA)8!ELR%EP#y4O<9Oy!hBCD;HVFU:%NrS=La_0 ^UfXp7zg߭-km$zEzEItSgW|ecR@UYTU3B;DY0ѣ`]/`6L녑3PvV?y 6J^L^/>]Jƾpɟa͵'pK,ߐ{iW1cx]zQ_7ʡv貘]UqR㕎!7;@[@5rrHK7f MM7f|4c;Sv߿eֲ׌nF'|w8٥rSջ[,1PLgo,{70,`/QjGT/Wa܊ L#]5[@ħD%X(O&:=ߝu^:W_-TB0K(^AE')p,Fͭ\oi[$I2sDiGZ)&\*`~uPUTNMx%emֹmD¤heѦ꼕K6Wζп PMlV8Ɉ:bM0LH3/3Ⱦ/L*ۦuL59X_XǞf X)[ 4d>#ى;%G HѠ7Nt~βpЃ!z}! +T@QX 6pA#*'YzagD J¥kH8p&_n_68/q紴!~OCcz7x Gf>'ƭS ݽ(m>zŁƍ# Q>`o [E +ĤNj2"нN^5${oZRfQJ4 )Ɏ vO!s&,6@Fms3%R\Rb/qH ,LjYʬl'\ 4lT2gʕs攔?]5s^þ7E۲-ƶI{ܶmS۶o~.葷A&[jwd72Ѽz!/@]Pk(P$( +:/U(ثRy xM\AU$-ST=(ƀ2ʁj +2x/\EfT@ dIYe<GGf6C7Zڭ$-{@<YRuUJ{UܡbzdmhE-B)Am.G~Wvͬ?>կVmzdrϮ#Yυ)^i@X4Xqa/:lh}?MAm!A:Y;T.jwww.hGnkm/0GQ( G4*yrG (+$]ts5΃^ez5=ήFQ }vGypZOe*O\f֙A L)wqD$ qRE.i:.Sr\e@H%5MM\vR|+lR<JZ5Umv*-lg!#]?E{8簓 +jk(H)*Gwd/:H(ETgC%;ᖣS%i7@Y Jd0RqLg"eID7%$9^F i_<"y2.ON03 03 eM8Oz¶r $[Z_'-/77g i"Y4ؔr% k&r_,6cx΀V:)[eJ UP0`V$ )R%&|x*%*F}DE9:.wJG]ΡYJW,æRC<Ȁ>ָf5qVȯ4֧A]yH@=@@>ebd+JULFޜ'S|%w$rWy)&d$ X*(<ϔ :AoaWĔb [I3-:-W Ui\B,)ݖ҈j-O06:]Wzk_Z[oo\Ņ8} ^}p![JZ\KX έb͎$ WYREUtcW+r2H-Ur. ߕ?}=„ݹi1ի7ٺg_W j4YV`F.0j3P|FZҨ !GKLR]9)-R!+~rXWnM: _C1^Iw(LD}]0PUR^AT!^DkN*HUk]uRƛΚc fsm&;SL: +qZڅM~<Xӧ-v[?qC3m]l&^ubn>LF^{juAdh &VtDV|Uuˮ*+~,Ty+ՉnU!S]ΉMM䱩 +Y.MusGQe$22紝9c1q'T_Y~6^ |{GΌs}F󿒓?.ncRj=q'̣"}&)`_&wdZSE,%#k6zhy>ڐuUj ^6i,}ouooBLI,i|&9fQOz̆9g>اA Ƈu6n.yݻuE u< [DG">@BGRT9iM ⎎r3L<,p 8[+&m~A OUGU +fT5!CDE2`/2 *WP <4 |QJ?K5 s/pq%MhgbT!cDSO^1QIyLҍf +J7]}aǐ":J)ÔQ"x ea }\X{\i 4ҸTtٽisV}ה Ҡ!w|t^CJ֫%@g|=܏N҃ ݥ]> a_BOlhW(]bNx}ID|Boz͊YpqRg駑E* HO—>eϲup|Uu9urh4~^p^|fPQ2rE7ʕu FK K1|<#Zg +3 uқ) kI#Κ3xO?"^wXE Km Ki2F&m#:GF=$_7jAP`d'l&%觟$_THYJw&7P=Iy/pv +7t( +fx%{k6M8:vᢲ=cH# - +>S_߹wme浫W^%ֲ~(;gb.|~ m\@O_ʼnT-uYAp9f$DF:Q2v&Vm\r>m >pNW'd~h + +=[15M:` ~0;"AGme!rwhju'{ ! I18w+W c;N~M=]S̠cJWnX7w Rsk;mws+SIΙ?>s 좟##:wڹOY>aen#lDAEhsLa^nm0lOB +|j0EC?9R4fYYV_{^!uI]Ն7Z*[_42~7CF!C H@$DTuşxyL8x>.za?yb#1B%"+ZM1M1 ^76UV,Doq|ӯ*n?}׫\-~ͲuSst=N Ich[0ˠÒ1[ •  pcaQP]J&"f)d:YL*JX&.%&rwqF g1 +sh?HPs\^*RRT}^h3ϕ.%B0yJ}EhUV^QYU5j{yNER vg ;a/{ Fs]j}aѲZW?-GaҤܼ +Wv:7U/VYY8-s!P#ܛ=]=sqĻ?x 3˟y|3Ϭ޼I/]Ϛ?}4N؍;F +#`\3|~TcQj.z۹ b{0rWͷWJxt})I>þxpj ul-^7{[#cv 9:w~hqwk̜:$7xq8]rW`\&QY1NU:JiE-sxg%N6a3r!Uiʦ]MdRħ„V}K_[w0Cі"h,|^ydn/{ɒ;wO.8'LB~=g &! Cd(bQ(D}dm6yE +UA$^BҸ#.8<vM[nh1`ܦSyfZY:Ir3pλ(9qD[=w] hH8H]6ģxܑċrɐC)GPr }B]aRmNf,qnqT|RQ}Y #<&FjI{bO&ę2.VK?+Tȭ-y.x;ױ[^߿omnא[II+~Ad  S'oiUNWA:'H6_,(0$X#qJ1S2dMn@;tBXIJP:t B G5h]N#vU(ZLt$qЍ=_d0Y0eZ*.:`h/3K^]c<t; ykЩb0!Ďbzf 䋂[~gBY?.;zz+s{\?|` lccVtqSŨ >!,b_+F"Ll,4HQ/߸h{/HQ~߳}ވPY;̂CQX]{'⤹72,a_Є;l +v q?=Ou柏I p.^?r̂cGs|gC_o*6]ъǪ~Y[AAݼQ u|ӻ wEB_`%D B=fh&z NDS!fAPʀ#=EPEd@l!wZ/44EZk3þTQo͆FpTw22C3e>=SYp_ <Ɲ]Z/Ap|(P9pZTd{o{YNw>Ac ?.qrvvB +#bhBp @E.{"a@$gT7½ q;~sv=~򣓧ԩKRg<:mJ;yA ?6ާW +endstream +endobj +1060 0 obj +<< +/Type /FontDescriptor +/FontName /SXAKMW+DejaVuSans +/FontFamily (DejaVu Sans) +/Flags 4 +/FontBBox [ 0 -14 557 560 ] +/ItalicAngle 0 +/Ascent 928 +/Descent -235 +/CapHeight 560 +/StemV 80 +/StemH 80 +/FontFile2 1056 0 R +>> +endobj +1061 0 obj +<< +/Type /Font +/Subtype /CIDFontType2 +/BaseFont /SXAKMW+DejaVuSans +/CIDSystemInfo << +/Registry (Adobe) +/Ordering (Identity) +/Supplement 0 +>> +/W [ 3 [ 317 ] 16 [ 360 317 ] 20 [ 636 636 ] 25 [ 636 ] 36 [ 684 686 698 ] 42 [ 774 ] 44 [ 294 ] 53 [ 694 634 ] 68 [ 612 ] 70 [ 549 634 615 352 ] 75 [ 633 277 ] 78 [ 579 277 974 633 611 ] 85 [ 411 520 392 633 591 ] ] +/FontDescriptor 1060 0 R +>> +endobj +1062 0 obj +<< +/Length 744 +>> +stream +/CIDInit /ProcSet findresource begin +12 dict begin +begincmap +/CIDSystemInfo +<< /Registry (Adobe) +/Ordering (UCS) +/Supplement 0 +>> def +/CMapName /Adobe-Identity-UCS def +/CMapType 2 def +1 begincodespacerange +<0000> +endcodespacerange +30 beginbfchar +<0026> <0043> +<002c> <0049> +<0036> <0053> +<0003> <0020> +<0059> <0076> +<0014> <0031> +<0011> <002e> +<0019> <0036> +<0025> <0042> +<0048> <0065> +<0051> <006e> +<0046> <0063> +<004b> <0068> +<0050> <006d> +<0044> <0061> +<0055> <0072> +<004e> <006b> +<0010> <002d> +<004f> <006c> +<0049> <0066> +<0024> <0041> +<0056> <0073> +<0057> <0074> +<002a> <0047> +<0058> <0075> +<004c> <0069> +<0047> <0064> +<0035> <0052> +<0015> <0032> +<0052> <006f> +endbfchar +endcmap +CMapName currentdict /CMap defineresource pop +end +end +endstream +endobj +1063 0 obj +<< +/Type /Font +/Subtype /Type0 +/BaseFont /SXAKMW+DejaVuSans +/Encoding /Identity-H +/DescendantFonts [ 1061 0 R ] +/ToUnicode 1062 0 R +>> +endobj +1064 0 obj +<< +/Type /FontDescriptor +/FontName /YFABGK+Poppins +/FontFamily (Poppins) +/Flags 4 +/FontBBox [ -27 -132 753 704 ] +/ItalicAngle 0 +/Ascent 1049 +/Descent -349 +/CapHeight 704 +/StemV 80 +/StemH 80 +/FontFile2 1057 0 R +>> +endobj +1065 0 obj +<< +/Type /Font +/Subtype /CIDFontType2 +/BaseFont /YFABGK+Poppins +/CIDSystemInfo << +/Registry (Adobe) +/Ordering (Identity) +/Supplement 0 +>> +/W [ 3 [ 266 ] 5 [ 291 ] 7 [ 621 ] 10 [ 158 453 453 485 ] 15 [ 197 550 209 475 627 319 574 588 628 627 634 545 630 629 212 ] 32 [ 722 ] 36 [ 673 612 771 706 512 503 777 691 245 ] 46 [ 598 431 860 702 785 578 787 607 586 540 674 675 975 ] 60 [ 583 540 422 ] 64 [ 422 ] 66 [ 732 256 675 675 606 675 619 328 675 639 245 247 514 245 1029 639 639 675 675 372 521 363 639 560 819 478 562 454 ] 95 [ 290 ] 204 [ 218 ] ] +/FontDescriptor 1064 0 R +>> +endobj +1066 0 obj +<< +/Length 1430 +>> +stream +/CIDInit /ProcSet findresource begin +12 dict begin +begincmap +/CIDSystemInfo +<< /Registry (Adobe) +/Ordering (UCS) +/Supplement 0 +>> def +/CMapName /Adobe-Identity-UCS def +/CMapType 2 def +1 begincodespacerange +<0000> +endcodespacerange +79 beginbfchar +<0026> <0043> +<002c> <0049> +<0036> <0053> +<0003> <0020> +<0059> <0076> +<0014> <0031> +<0011> <002e> +<0019> <0036> +<0025> <0042> +<0048> <0065> +<0051> <006e> +<0046> <0063> +<004b> <0068> +<0050> <006d> +<0044> <0061> +<0055> <0072> +<004e> <006b> +<0010> <002d> +<004f> <006c> +<0049> <0066> +<0024> <0041> +<0056> <0073> +<0057> <0074> +<002a> <0047> +<0058> <0075> +<004c> <0069> +<0047> <0064> +<0035> <0052> +<0015> <0032> +<0016> <0033> +<002e> <004b> +<0045> <0062> +<005a> <0077> +<001b> <0038> +<0052> <006f> +<0030> <004d> +<0031> <004e> +<004a> <0067> +<0029> <0046> +<0017> <0034> +<0033> <0050> +<001a> <0037> +<0018> <0035> +<0013> <0030> +<0028> <0045> +<005d> <007a> +<002f> <004c> +<003a> <0057> +<005c> <0079> +<001c> <0039> +<005b> <0078> +<0027> <0044> +<0032> <004f> +<0037> <0054> +<0053> <0070> +<000f> <002c> +<001d> <003a> +<002b> <0048> +<0039> <0056> +<003c> <0059> +<000b> <0028> +<000c> <0029> +<004d> <006a> +<0054> <0071> +<000a> <0027> +<005f> <007c> +<0012> <002f> +<000d> <002a> +<0020> <003d> +<0042> <005f> +<0043> <0060> +<0005> <0022> +<003e> <005b> +<0040> <005d> +<00cc> <2019> +<0007> <0024> +<0038> <0055> +<003d> <005a> +<0034> <0051> +endbfchar +endcmap +CMapName currentdict /CMap defineresource pop +end +end +endstream +endobj +1067 0 obj +<< +/Type /Font +/Subtype /Type0 +/BaseFont /YFABGK+Poppins +/Encoding /Identity-H +/DescendantFonts [ 1065 0 R ] +/ToUnicode 1066 0 R +>> +endobj +1068 0 obj +<< +/Type /FontDescriptor +/FontName /PPVKZX+DejaVuSans +/FontFamily (DejaVu Sans) +/Flags 4 +/FontBBox [ -9 -92 733 729 ] +/ItalicAngle 0 +/Ascent 928 +/Descent -235 +/CapHeight 729 +/StemV 80 +/StemH 80 +/FontFile2 1058 0 R +>> +endobj +1069 0 obj +<< +/Type /Font +/Subtype /CIDFontType2 +/BaseFont /PPVKZX+DejaVuSans +/CIDSystemInfo << +/Registry (Adobe) +/Ordering (Identity) +/Supplement 0 +>> +/W [ 3 [ 348 ] 11 [ 457 457 ] 16 [ 415 379 365 695 695 695 695 695 695 695 695 695 695 399 ] 36 [ 773 762 733 830 683 ] 44 [ 372 ] 46 [ 774 637 995 836 ] 51 [ 732 ] 53 [ 770 720 682 ] 57 [ 773 1103 ] 60 [ 724 ] 66 [ 500 ] 68 [ 674 715 592 715 678 435 715 711 342 ] 78 [ 665 342 1041 711 687 715 715 493 595 478 711 651 923 645 651 582 ] 5038 [ 741 ] ] +/FontDescriptor 1068 0 R +>> +endobj +1070 0 obj +<< +/Length 1182 +>> +stream +/CIDInit /ProcSet findresource begin +12 dict begin +begincmap +/CIDSystemInfo +<< /Registry (Adobe) +/Ordering (UCS) +/Supplement 0 +>> def +/CMapName /Adobe-Identity-UCS def +/CMapType 2 def +1 begincodespacerange +<0000> +endcodespacerange +61 beginbfchar +<0014> <0031> +<0016> <0033> +<0017> <0034> +<0018> <0035> +<0011> <002e> +<0003> <0020> +<0028> <0045> +<0051> <006e> +<0056> <0073> +<0058> <0075> +<0055> <0072> +<0048> <0065> +<0057> <0074> +<004b> <0068> +<0044> <0061> +<0024> <0041> +<0033> <0050> +<002c> <0049> +<0059> <0076> +<0053> <0070> +<0052> <006f> +<0047> <0064> +<0046> <0063> +<004c> <0069> +<13ae> <00660069> +<004f> <006c> +<0050> <006d> +<0019> <0036> +<000b> <0028> +<000c> <0029> +<0015> <0032> +<005a> <0077> +<001d> <003a> +<004a> <0067> +<001a> <0037> +<001b> <0038> +<001c> <0039> +<0026> <0043> +<0031> <004e> +<004e> <006b> +<0049> <0066> +<0030> <004d> +<0013> <0030> +<005c> <0079> +<0010> <002d> +<002e> <004b> +<0045> <0062> +<005d> <007a> +<0035> <0052> +<0025> <0042> +<002f> <004c> +<0036> <0053> +<005b> <0078> +<0027> <0044> +<0054> <0071> +<0039> <0056> +<0037> <0054> +<0042> <005f> +<003a> <0057> +<0012> <002f> +<003c> <0059> +endbfchar +endcmap +CMapName currentdict /CMap defineresource pop +end +end +endstream +endobj +1071 0 obj +<< +/Type /Font +/Subtype /Type0 +/BaseFont /PPVKZX+DejaVuSans +/Encoding /Identity-H +/DescendantFonts [ 1069 0 R ] +/ToUnicode 1070 0 R +>> +endobj +1072 0 obj +<< +/Type /FontDescriptor +/FontName /MWAATU+DejaVuSansMono +/FontFamily (DejaVu Sans Mono) +/Flags 4 +/FontBBox [ 0 -155 598 728 ] +/ItalicAngle 0 +/Ascent 928 +/Descent -235 +/CapHeight 728 +/StemV 80 +/StemH 80 +/FontFile2 1059 0 R +>> +endobj +1073 0 obj +<< +/Type /Font +/Subtype /CIDFontType2 +/BaseFont /MWAATU+DejaVuSansMono +/CIDSystemInfo << +/Registry (Adobe) +/Ordering (Identity) +/Supplement 0 +>> +/W [ 3 [ 602 602 602 602 602 602 602 602 602 602 602 ] 15 [ 602 602 601 602 602 602 602 602 602 602 602 602 602 602 602 602 602 602 602 602 602 601 602 602 602 602 602 602 602 602 ] 46 [ 602 602 602 601 602 602 602 602 602 602 602 602 602 602 602 ] 62 [ 602 602 602 ] 66 [ 601 ] 68 [ 601 601 601 601 601 602 602 602 601 602 601 601 601 602 601 601 602 601 601 601 601 602 601 602 602 602 602 602 602 ] ] +/FontDescriptor 1072 0 R +>> +endobj +1074 0 obj +<< +/Length 1570 +>> +stream +/CIDInit /ProcSet findresource begin +12 dict begin +begincmap +/CIDSystemInfo +<< /Registry (Adobe) +/Ordering (UCS) +/Supplement 0 +>> def +/CMapName /Adobe-Identity-UCS def +/CMapType 2 def +1 begincodespacerange +<0000> +endcodespacerange +89 beginbfchar +<0031> <004e> +<0052> <006f> +<0057> <0074> +<0024> <0041> +<0053> <0070> +<004f> <006c> +<004c> <0069> +<0046> <0063> +<0044> <0061> +<0045> <0062> +<0048> <0065> +<0058> <0075> +<0050> <006d> +<0047> <0064> +<0056> <0073> +<0055> <0072> +<0003> <0020> +<0010> <002d> +<0051> <006e> +<0020> <003d> +<0008> <0025> +<001f> <003c> +<004b> <0068> +<0012> <002f> +<0049> <0066> +<0021> <003e> +<0038> <0055> +<001d> <003a> +<002a> <0047> +<0059> <0076> +<000a> <0027> +<001a> <0037> +<0013> <0030> +<0054> <0071> +<004e> <006b> +<0042> <005f> +<005a> <0077> +<0011> <002e> +<0006> <0023> +<0004> <0021> +<0037> <0054> +<004a> <0067> +<005c> <0079> +<0007> <0024> +<0014> <0031> +<002c> <0049> +<0033> <0050> +<0027> <0044> +<0035> <0052> +<003e> <005b> +<0005> <0022> +<005e> <007b> +<0060> <007d> +<0040> <005d> +<001e> <003b> +<005b> <0078> +<000b> <0028> +<000c> <0029> +<002f> <004c> +<000d> <002a> +<005f> <007c> +<0015> <0032> +<0009> <0026> +<0019> <0036> +<0017> <0034> +<002b> <0048> +<0036> <0053> +<0028> <0045> +<0030> <004d> +<0032> <004f> +<0029> <0046> +<003f> <005c> +<0022> <003f> +<005d> <007a> +<0016> <0033> +<003b> <0058> +<000f> <002c> +<0025> <0042> +<0026> <0043> +<001c> <0039> +<003a> <0057> +<001b> <0038> +<0018> <0035> +<003c> <0059> +<004d> <006a> +<0039> <0056> +<0034> <0051> +<0023> <0040> +<002e> <004b> +endbfchar +endcmap +CMapName currentdict /CMap defineresource pop +end +end +endstream +endobj +1075 0 obj +<< +/Type /Font +/Subtype /Type0 +/BaseFont /MWAATU+DejaVuSansMono +/Encoding /Identity-H +/DescendantFonts [ 1073 0 R ] +/ToUnicode 1074 0 R +>> +endobj +1076 0 obj +<< +/SXAKMW 1063 0 R +/YFABGK 1067 0 R +/PPVKZX 1071 0 R +/MWAATU 1075 0 R +>> +endobj +xref +0 1077 +0000000000 65535 f +0000000015 00000 n +0000001112 00000 n +0000001246 00000 n +0000040647 00000 n +0000040781 00000 n +0000043473 00000 n +0000043672 00000 n +0000047406 00000 n +0000047812 00000 n +0000048013 00000 n +0000048216 00000 n +0000048351 00000 n +0000048487 00000 n +0000048645 00000 n +0000048804 00000 n +0000049037 00000 n +0000049271 00000 n +0000049486 00000 n +0000049702 00000 n +0000049943 00000 n +0000050185 00000 n +0000050408 00000 n +0000050632 00000 n +0000050864 00000 n +0000051097 00000 n +0000051311 00000 n +0000051526 00000 n +0000051753 00000 n +0000051981 00000 n +0000052190 00000 n +0000052400 00000 n +0000052629 00000 n +0000052859 00000 n +0000053070 00000 n +0000053282 00000 n +0000053501 00000 n +0000053720 00000 n +0000057475 00000 n +0000057940 00000 n +0000058141 00000 n +0000058343 00000 n +0000058558 00000 n +0000058774 00000 n +0000058971 00000 n +0000059169 00000 n +0000059388 00000 n +0000059608 00000 n +0000059809 00000 n +0000060011 00000 n +0000060239 00000 n +0000060468 00000 n +0000060678 00000 n +0000060889 00000 n +0000061104 00000 n +0000061320 00000 n +0000061551 00000 n +0000061783 00000 n +0000061986 00000 n +0000062190 00000 n +0000062327 00000 n +0000062465 00000 n +0000062659 00000 n +0000062854 00000 n +0000063044 00000 n +0000063235 00000 n +0000063426 00000 n +0000063618 00000 n +0000063810 00000 n +0000064003 00000 n +0000064242 00000 n +0000064482 00000 n +0000064697 00000 n +0000064913 00000 n +0000065120 00000 n +0000065327 00000 n +0000068893 00000 n +0000069371 00000 n +0000069567 00000 n +0000069764 00000 n +0000069960 00000 n +0000070157 00000 n +0000070356 00000 n +0000070556 00000 n +0000070756 00000 n +0000070957 00000 n +0000071155 00000 n +0000071354 00000 n +0000071588 00000 n +0000071823 00000 n +0000072022 00000 n +0000072222 00000 n +0000072425 00000 n +0000072629 00000 n +0000072831 00000 n +0000073034 00000 n +0000073234 00000 n +0000073435 00000 n +0000073631 00000 n +0000073828 00000 n +0000074017 00000 n +0000074208 00000 n +0000074400 00000 n +0000074593 00000 n +0000074783 00000 n +0000074974 00000 n +0000075160 00000 n +0000075347 00000 n +0000075559 00000 n +0000075772 00000 n +0000075987 00000 n +0000076203 00000 n +0000076416 00000 n +0000076629 00000 n +0000080328 00000 n +0000080831 00000 n +0000081033 00000 n +0000081236 00000 n +0000081438 00000 n +0000081641 00000 n +0000081852 00000 n +0000082064 00000 n +0000082285 00000 n +0000082507 00000 n +0000082736 00000 n +0000082966 00000 n +0000083167 00000 n +0000083369 00000 n +0000083567 00000 n +0000083766 00000 n +0000083979 00000 n +0000084193 00000 n +0000084392 00000 n +0000084592 00000 n +0000084803 00000 n +0000085015 00000 n +0000085161 00000 n +0000085308 00000 n +0000085522 00000 n +0000085737 00000 n +0000085929 00000 n +0000086122 00000 n +0000086333 00000 n +0000086545 00000 n +0000086764 00000 n +0000086984 00000 n +0000087183 00000 n +0000087383 00000 n +0000087591 00000 n +0000087800 00000 n +0000087996 00000 n +0000088192 00000 n +0000092056 00000 n +0000092527 00000 n +0000092664 00000 n +0000092802 00000 n +0000092994 00000 n +0000093187 00000 n +0000093386 00000 n +0000093586 00000 n +0000093743 00000 n +0000093901 00000 n +0000094114 00000 n +0000094328 00000 n +0000094524 00000 n +0000094721 00000 n +0000094913 00000 n +0000095106 00000 n +0000095329 00000 n +0000095553 00000 n +0000095754 00000 n +0000095956 00000 n +0000096153 00000 n +0000096351 00000 n +0000096548 00000 n +0000096746 00000 n +0000096906 00000 n +0000097067 00000 n +0000097264 00000 n +0000097462 00000 n +0000097597 00000 n +0000097733 00000 n +0000097916 00000 n +0000098100 00000 n +0000098292 00000 n +0000098485 00000 n +0000102326 00000 n +0000102813 00000 n +0000102972 00000 n +0000103132 00000 n +0000103353 00000 n +0000103575 00000 n +0000103778 00000 n +0000103982 00000 n +0000104205 00000 n +0000104429 00000 n +0000104633 00000 n +0000104838 00000 n +0000105069 00000 n +0000105301 00000 n +0000105514 00000 n +0000105728 00000 n +0000105957 00000 n +0000106187 00000 n +0000106405 00000 n +0000106624 00000 n +0000106860 00000 n +0000107097 00000 n +0000107315 00000 n +0000107534 00000 n +0000107669 00000 n +0000107805 00000 n +0000107998 00000 n +0000108192 00000 n +0000108403 00000 n +0000108615 00000 n +0000108818 00000 n +0000109022 00000 n +0000109213 00000 n +0000109405 00000 n +0000109619 00000 n +0000109833 00000 n +0000113909 00000 n +0000114396 00000 n +0000114599 00000 n +0000114803 00000 n +0000115008 00000 n +0000115214 00000 n +0000115404 00000 n +0000115595 00000 n +0000115832 00000 n +0000116070 00000 n +0000116301 00000 n +0000116533 00000 n +0000116737 00000 n +0000116942 00000 n +0000117150 00000 n +0000117359 00000 n +0000117567 00000 n +0000117776 00000 n +0000117929 00000 n +0000118083 00000 n +0000118280 00000 n +0000118478 00000 n +0000118639 00000 n +0000118801 00000 n +0000118983 00000 n +0000119166 00000 n +0000119331 00000 n +0000119497 00000 n +0000119694 00000 n +0000119892 00000 n +0000120094 00000 n +0000120297 00000 n +0000120446 00000 n +0000120596 00000 n +0000120778 00000 n +0000120961 00000 n +0000124800 00000 n +0000125271 00000 n +0000125492 00000 n +0000125714 00000 n +0000125928 00000 n +0000126143 00000 n +0000126361 00000 n +0000126580 00000 n +0000126784 00000 n +0000126989 00000 n +0000127165 00000 n +0000127342 00000 n +0000127541 00000 n +0000127741 00000 n +0000127936 00000 n +0000128132 00000 n +0000128330 00000 n +0000128529 00000 n +0000128681 00000 n +0000128834 00000 n +0000129021 00000 n +0000129209 00000 n +0000129403 00000 n +0000129598 00000 n +0000129744 00000 n +0000129891 00000 n +0000130093 00000 n +0000130296 00000 n +0000130463 00000 n +0000130631 00000 n +0000130787 00000 n +0000130944 00000 n +0000131151 00000 n +0000131359 00000 n +0000133666 00000 n +0000133961 00000 n +0000134105 00000 n +0000134250 00000 n +0000134452 00000 n +0000134655 00000 n +0000134869 00000 n +0000135084 00000 n +0000135269 00000 n +0000135455 00000 n +0000135633 00000 n +0000135812 00000 n +0000140610 00000 n +0000140849 00000 n +0000141105 00000 n +0000141312 00000 n +0000141518 00000 n +0000145091 00000 n +0000145322 00000 n +0000145519 00000 n +0000145708 00000 n +0000149393 00000 n +0000149596 00000 n +0000153335 00000 n +0000153558 00000 n +0000153692 00000 n +0000157714 00000 n +0000157953 00000 n +0000158087 00000 n +0000158221 00000 n +0000158355 00000 n +0000161968 00000 n +0000162171 00000 n +0000166877 00000 n +0000167268 00000 n +0000167402 00000 n +0000167534 00000 n +0000167666 00000 n +0000167798 00000 n +0000167930 00000 n +0000168062 00000 n +0000168194 00000 n +0000168326 00000 n +0000168458 00000 n +0000168591 00000 n +0000168724 00000 n +0000168857 00000 n +0000168990 00000 n +0000169123 00000 n +0000169256 00000 n +0000169389 00000 n +0000169522 00000 n +0000169655 00000 n +0000169788 00000 n +0000169921 00000 n +0000170053 00000 n +0000170184 00000 n +0000175185 00000 n +0000175584 00000 n +0000175717 00000 n +0000175850 00000 n +0000175983 00000 n +0000176116 00000 n +0000176249 00000 n +0000176382 00000 n +0000176515 00000 n +0000176648 00000 n +0000176781 00000 n +0000176914 00000 n +0000177047 00000 n +0000177180 00000 n +0000177313 00000 n +0000177446 00000 n +0000177579 00000 n +0000177712 00000 n +0000177845 00000 n +0000177978 00000 n +0000178111 00000 n +0000178244 00000 n +0000178377 00000 n +0000178510 00000 n +0000178645 00000 n +0000183449 00000 n +0000183912 00000 n +0000184047 00000 n +0000184180 00000 n +0000184313 00000 n +0000184446 00000 n +0000184579 00000 n +0000184712 00000 n +0000184845 00000 n +0000184978 00000 n +0000185111 00000 n +0000185245 00000 n +0000185379 00000 n +0000185513 00000 n +0000185647 00000 n +0000185781 00000 n +0000185915 00000 n +0000186049 00000 n +0000186183 00000 n +0000186317 00000 n +0000186451 00000 n +0000186585 00000 n +0000186719 00000 n +0000186853 00000 n +0000186987 00000 n +0000187121 00000 n +0000187255 00000 n +0000187389 00000 n +0000187523 00000 n +0000187657 00000 n +0000187791 00000 n +0000187925 00000 n +0000188058 00000 n +0000193055 00000 n +0000193518 00000 n +0000193652 00000 n +0000193786 00000 n +0000193920 00000 n +0000194054 00000 n +0000194188 00000 n +0000194322 00000 n +0000194456 00000 n +0000194590 00000 n +0000194724 00000 n +0000194858 00000 n +0000194992 00000 n +0000195126 00000 n +0000195260 00000 n +0000195394 00000 n +0000195528 00000 n +0000195662 00000 n +0000195796 00000 n +0000195930 00000 n +0000196064 00000 n +0000196198 00000 n +0000196332 00000 n +0000196466 00000 n +0000196600 00000 n +0000196734 00000 n +0000196868 00000 n +0000197002 00000 n +0000197136 00000 n +0000197270 00000 n +0000197404 00000 n +0000197538 00000 n +0000197673 00000 n +0000202259 00000 n +0000202650 00000 n +0000202785 00000 n +0000202918 00000 n +0000203051 00000 n +0000203184 00000 n +0000203317 00000 n +0000203450 00000 n +0000203583 00000 n +0000203716 00000 n +0000203849 00000 n +0000203983 00000 n +0000204117 00000 n +0000204251 00000 n +0000204385 00000 n +0000204519 00000 n +0000204653 00000 n +0000204787 00000 n +0000204921 00000 n +0000205055 00000 n +0000205189 00000 n +0000205323 00000 n +0000205457 00000 n +0000205590 00000 n +0000210570 00000 n +0000211081 00000 n +0000211215 00000 n +0000211349 00000 n +0000211483 00000 n +0000211617 00000 n +0000211751 00000 n +0000211885 00000 n +0000212019 00000 n +0000212153 00000 n +0000212287 00000 n +0000212421 00000 n +0000212555 00000 n +0000212689 00000 n +0000212823 00000 n +0000212957 00000 n +0000213091 00000 n +0000213225 00000 n +0000213359 00000 n +0000213493 00000 n +0000213627 00000 n +0000213761 00000 n +0000213895 00000 n +0000214029 00000 n +0000214163 00000 n +0000214297 00000 n +0000214431 00000 n +0000214565 00000 n +0000214699 00000 n +0000214833 00000 n +0000214967 00000 n +0000215101 00000 n +0000215235 00000 n +0000215369 00000 n +0000215503 00000 n +0000215637 00000 n +0000215771 00000 n +0000215904 00000 n +0000216036 00000 n +0000219278 00000 n +0000219517 00000 n +0000219651 00000 n +0000219785 00000 n +0000219920 00000 n +0000224952 00000 n +0000225175 00000 n +0000225310 00000 n +0000230303 00000 n +0000230526 00000 n +0000230661 00000 n +0000235447 00000 n +0000235650 00000 n +0000240649 00000 n +0000240872 00000 n +0000241007 00000 n +0000245852 00000 n +0000246055 00000 n +0000250970 00000 n +0000251193 00000 n +0000251328 00000 n +0000256189 00000 n +0000256412 00000 n +0000256547 00000 n +0000261310 00000 n +0000261513 00000 n +0000266505 00000 n +0000266728 00000 n +0000266863 00000 n +0000271838 00000 n +0000272041 00000 n +0000276936 00000 n +0000277159 00000 n +0000277294 00000 n +0000282231 00000 n +0000282454 00000 n +0000282589 00000 n +0000287403 00000 n +0000287606 00000 n +0000292603 00000 n +0000292826 00000 n +0000292961 00000 n +0000297965 00000 n +0000298188 00000 n +0000298323 00000 n +0000303231 00000 n +0000303434 00000 n +0000308403 00000 n +0000308626 00000 n +0000308761 00000 n +0000313532 00000 n +0000313735 00000 n +0000318047 00000 n +0000318278 00000 n +0000318413 00000 n +0000318548 00000 n +0000323431 00000 n +0000323654 00000 n +0000323789 00000 n +0000328814 00000 n +0000329037 00000 n +0000329172 00000 n +0000334040 00000 n +0000334243 00000 n +0000339339 00000 n +0000339562 00000 n +0000339697 00000 n +0000344581 00000 n +0000344784 00000 n +0000349796 00000 n +0000350019 00000 n +0000350154 00000 n +0000355070 00000 n +0000355293 00000 n +0000355428 00000 n +0000360228 00000 n +0000360431 00000 n +0000365468 00000 n +0000365691 00000 n +0000365826 00000 n +0000370748 00000 n +0000370971 00000 n +0000371106 00000 n +0000375917 00000 n +0000376120 00000 n +0000381088 00000 n +0000381311 00000 n +0000381446 00000 n +0000386387 00000 n +0000386590 00000 n +0000391465 00000 n +0000391688 00000 n +0000391823 00000 n +0000396759 00000 n +0000396982 00000 n +0000397117 00000 n +0000401918 00000 n +0000402121 00000 n +0000407114 00000 n +0000407337 00000 n +0000407472 00000 n +0000412461 00000 n +0000412684 00000 n +0000412819 00000 n +0000417598 00000 n +0000417801 00000 n +0000422784 00000 n +0000423007 00000 n +0000423142 00000 n +0000427990 00000 n +0000428193 00000 n +0000433212 00000 n +0000433435 00000 n +0000433570 00000 n +0000438436 00000 n +0000438659 00000 n +0000438794 00000 n +0000443530 00000 n +0000443733 00000 n +0000448788 00000 n +0000449011 00000 n +0000449147 00000 n +0000454091 00000 n +0000454294 00000 n +0000459120 00000 n +0000459343 00000 n +0000459479 00000 n +0000464495 00000 n +0000464718 00000 n +0000464854 00000 n +0000469631 00000 n +0000469834 00000 n +0000474808 00000 n +0000475031 00000 n +0000475167 00000 n +0000480155 00000 n +0000480378 00000 n +0000480513 00000 n +0000485271 00000 n +0000485474 00000 n +0000490551 00000 n +0000490886 00000 n +0000491022 00000 n +0000491156 00000 n +0000491290 00000 n +0000491424 00000 n +0000491558 00000 n +0000491692 00000 n +0000491826 00000 n +0000491960 00000 n +0000492094 00000 n +0000492229 00000 n +0000492364 00000 n +0000492499 00000 n +0000492634 00000 n +0000492768 00000 n +0000492901 00000 n +0000497397 00000 n +0000497708 00000 n +0000497843 00000 n +0000497978 00000 n +0000498113 00000 n +0000498248 00000 n +0000498383 00000 n +0000498518 00000 n +0000498653 00000 n +0000498788 00000 n +0000498923 00000 n +0000499058 00000 n +0000499194 00000 n +0000499330 00000 n +0000504338 00000 n +0000504561 00000 n +0000504697 00000 n +0000509536 00000 n +0000509759 00000 n +0000509895 00000 n +0000514751 00000 n +0000514974 00000 n +0000515110 00000 n +0000519926 00000 n +0000520149 00000 n +0000520285 00000 n +0000525136 00000 n +0000525359 00000 n +0000525495 00000 n +0000530166 00000 n +0000530389 00000 n +0000530525 00000 n +0000534114 00000 n +0000534317 00000 n +0000538901 00000 n +0000539132 00000 n +0000539268 00000 n +0000539403 00000 n +0000542898 00000 n +0000543101 00000 n +0000547952 00000 n +0000548175 00000 n +0000548311 00000 n +0000553075 00000 n +0000553278 00000 n +0000558388 00000 n +0000558611 00000 n +0000558747 00000 n +0000563492 00000 n +0000563695 00000 n +0000568822 00000 n +0000569045 00000 n +0000569180 00000 n +0000573906 00000 n +0000574109 00000 n +0000579103 00000 n +0000579306 00000 n +0000584107 00000 n +0000584330 00000 n +0000584466 00000 n +0000589217 00000 n +0000589420 00000 n +0000594467 00000 n +0000594690 00000 n +0000594826 00000 n +0000599592 00000 n +0000599795 00000 n +0000604796 00000 n +0000605019 00000 n +0000605155 00000 n +0000609920 00000 n +0000610123 00000 n +0000614957 00000 n +0000615160 00000 n +0000619972 00000 n +0000620195 00000 n +0000620331 00000 n +0000625074 00000 n +0000625277 00000 n +0000628895 00000 n +0000629098 00000 n +0000632371 00000 n +0000632574 00000 n +0000637611 00000 n +0000637834 00000 n +0000637970 00000 n +0000642503 00000 n +0000642706 00000 n +0000646992 00000 n +0000647215 00000 n +0000647351 00000 n +0000651417 00000 n +0000651648 00000 n +0000651784 00000 n +0000651920 00000 n +0000656866 00000 n +0000657137 00000 n +0000657273 00000 n +0000657409 00000 n +0000657543 00000 n +0000657677 00000 n +0000657811 00000 n +0000657945 00000 n +0000658081 00000 n +0000662852 00000 n +0000663115 00000 n +0000663251 00000 n +0000663385 00000 n +0000663519 00000 n +0000663653 00000 n +0000663787 00000 n +0000663923 00000 n +0000666684 00000 n +0000666887 00000 n +0000671193 00000 n +0000671432 00000 n +0000671568 00000 n +0000671704 00000 n +0000671839 00000 n +0000675857 00000 n +0000676096 00000 n +0000676232 00000 n +0000676368 00000 n +0000676504 00000 n +0000681153 00000 n +0000681400 00000 n +0000681536 00000 n +0000681672 00000 n +0000681808 00000 n +0000681944 00000 n +0000686819 00000 n +0000687050 00000 n +0000687186 00000 n +0000687322 00000 n +0000691466 00000 n +0000691697 00000 n +0000691833 00000 n +0000691969 00000 n +0000696136 00000 n +0000696383 00000 n +0000696519 00000 n +0000696655 00000 n +0000696791 00000 n +0000696927 00000 n +0000701916 00000 n +0000702147 00000 n +0000702283 00000 n +0000702419 00000 n +0000706998 00000 n +0000707229 00000 n +0000707365 00000 n +0000707501 00000 n +0000710438 00000 n +0000710661 00000 n +0000710797 00000 n +0000714870 00000 n +0000715073 00000 n +0000720577 00000 n +0000720976 00000 n +0000721112 00000 n +0000721246 00000 n +0000721380 00000 n +0000721514 00000 n +0000721648 00000 n +0000721782 00000 n +0000721916 00000 n +0000722050 00000 n +0000722184 00000 n +0000722319 00000 n +0000722454 00000 n +0000722589 00000 n +0000722724 00000 n +0000722859 00000 n +0000722994 00000 n +0000723129 00000 n +0000723264 00000 n +0000723399 00000 n +0000723534 00000 n +0000723669 00000 n +0000723804 00000 n +0000723939 00000 n +0000724073 00000 n +0000727769 00000 n +0000728056 00000 n +0000728191 00000 n +0000728326 00000 n +0000728461 00000 n +0000728596 00000 n +0000728731 00000 n +0000728866 00000 n +0000729001 00000 n +0000729136 00000 n +0000729272 00000 n +0000733691 00000 n +0000733914 00000 n +0000734050 00000 n +0000738503 00000 n +0000738734 00000 n +0000738870 00000 n +0000739006 00000 n +0000743465 00000 n +0000743688 00000 n +0000743824 00000 n +0000748783 00000 n +0000749118 00000 n +0000749254 00000 n +0000749388 00000 n +0000749522 00000 n +0000749656 00000 n +0000749790 00000 n +0000749924 00000 n +0000750058 00000 n +0000750192 00000 n +0000750326 00000 n +0000750461 00000 n +0000750596 00000 n +0000750731 00000 n +0000750866 00000 n +0000751000 00000 n +0000751133 00000 n +0000754374 00000 n +0000754629 00000 n +0000754764 00000 n +0000754899 00000 n +0000755034 00000 n +0000755169 00000 n +0000755305 00000 n +0000758716 00000 n +0000758919 00000 n +0000762176 00000 n +0000762379 00000 n +0000766780 00000 n +0000766983 00000 n +0000771375 00000 n +0000771742 00000 n +0000771878 00000 n +0000772012 00000 n +0000772146 00000 n +0000772280 00000 n +0000772414 00000 n +0000772548 00000 n +0000772682 00000 n +0000772816 00000 n +0000772950 00000 n +0000773085 00000 n +0000773220 00000 n +0000773355 00000 n +0000773490 00000 n +0000773625 00000 n +0000773760 00000 n +0000773895 00000 n +0000774030 00000 n +0000774165 00000 n +0000774301 00000 n +0000774493 00000 n +0000774715 00000 n +0000774841 00000 n +0000774987 00000 n +0000775127 00000 n +0000775324 00000 n +0000775558 00000 n +0000775789 00000 n +0000776045 00000 n +0000776284 00000 n +0000776531 00000 n +0000776761 00000 n +0000777003 00000 n +0000777228 00000 n +0000777472 00000 n +0000777700 00000 n +0000777935 00000 n +0000778153 00000 n +0000778384 00000 n +0000778598 00000 n +0000778833 00000 n +0000779051 00000 n +0000779295 00000 n +0000779522 00000 n +0000779754 00000 n +0000780001 00000 n +0000780206 00000 n +0000780382 00000 n +0000780577 00000 n +0000780782 00000 n +0000780988 00000 n +0000781195 00000 n +0000781449 00000 n +0000781679 00000 n +0000781902 00000 n +0000782113 00000 n +0000782324 00000 n +0000782539 00000 n +0000782755 00000 n +0000782969 00000 n +0000783219 00000 n +0000783434 00000 n +0000783653 00000 n +0000783871 00000 n +0000784087 00000 n +0000784299 00000 n +0000784504 00000 n +0000784711 00000 n +0000784916 00000 n +0000785117 00000 n +0000785344 00000 n +0000785574 00000 n +0000785803 00000 n +0000786020 00000 n +0000786237 00000 n +0000786463 00000 n +0000786699 00000 n +0000786943 00000 n +0000787159 00000 n +0000787372 00000 n +0000787600 00000 n +0000787814 00000 n +0000788026 00000 n +0000788209 00000 n +0000788423 00000 n +0000788627 00000 n +0000788852 00000 n +0000789085 00000 n +0000789298 00000 n +0000789520 00000 n +0000789717 00000 n +0000789891 00000 n +0000790081 00000 n +0000790278 00000 n +0000790470 00000 n +0000790681 00000 n +0000790889 00000 n +0000791093 00000 n +0000791328 00000 n +0000791541 00000 n +0000791750 00000 n +0000791945 00000 n +0000792142 00000 n +0000792325 00000 n +0000792498 00000 n +0000792681 00000 n +0000792873 00000 n +0000793074 00000 n +0000793298 00000 n +0000793520 00000 n +0000793761 00000 n +0000793984 00000 n +0000794233 00000 n +0000794465 00000 n +0000794712 00000 n +0000794949 00000 n +0000795203 00000 n +0000795426 00000 n +0000795604 00000 n +0000795800 00000 n +0000796027 00000 n +0000796246 00000 n +0000796455 00000 n +0000796687 00000 n +0000796908 00000 n +0000797131 00000 n +0000797339 00000 n +0000797594 00000 n +0000797842 00000 n +0000798065 00000 n +0000798292 00000 n +0000798504 00000 n +0000798699 00000 n +0000798899 00000 n +0000799078 00000 n +0000799278 00000 n +0000799461 00000 n +0000799676 00000 n +0000799881 00000 n +0000800072 00000 n +0000800257 00000 n +0000800496 00000 n +0000800728 00000 n +0000800964 00000 n +0000801186 00000 n +0000801380 00000 n +0000801597 00000 n +0000801810 00000 n +0000802011 00000 n +0000802205 00000 n +0000802395 00000 n +0000802592 00000 n +0000802780 00000 n +0000802985 00000 n +0000803155 00000 n +0000803353 00000 n +0000803548 00000 n +0000803719 00000 n +0000803924 00000 n +0000804157 00000 n +0000804360 00000 n +0000804541 00000 n +0000804605 00000 n +0000812285 00000 n +0000817087 00000 n +0000828947 00000 n +0000841343 00000 n +0000841577 00000 n +0000841981 00000 n +0000842779 00000 n +0000842932 00000 n +0000843163 00000 n +0000843760 00000 n +0000845245 00000 n +0000845395 00000 n +0000845630 00000 n +0000846169 00000 n +0000847406 00000 n +0000847559 00000 n +0000847803 00000 n +0000848398 00000 n +0000850023 00000 n +0000850180 00000 n +trailer +<< +/Size 1077 +/Root 3 0 R +/Info 2 0 R +>> +startxref +850272 +%%EOF diff --git a/content/rancher/v2.6/en/security/hardening-guides/1.6-benchmark-2.6/_index.md b/content/rancher/v2.6/en/security/hardening-guides/1.6-benchmark-2.6/_index.md new file mode 100644 index 00000000000..b1eba5740d7 --- /dev/null +++ b/content/rancher/v2.6/en/security/hardening-guides/1.6-benchmark-2.6/_index.md @@ -0,0 +1,3098 @@ +--- +title: CIS v1.6 Benchmark - Self-Assessment Guide - Rancher v2.6 +weight: 101 +--- + +### CIS v1.6 Kubernetes Benchmark - Rancher v2.6 with Kubernetes v1.18 to v1.21 + +[Click here to download a PDF version of this document](https://releases.rancher.com/documents/security/2.6/Rancher_v2-6_CIS_v1-6_Benchmark_Assessment.pdf). + +#### Overview + +This document is a companion to the Rancher v2.6 security hardening guide. The hardening guide provides prescriptive guidance for hardening a production installation of Rancher, and this benchmark guide is meant to help you evaluate the level of security of the hardened cluster against each control in the benchmark. + +This guide corresponds to specific versions of the hardening guide, Rancher, CIS Benchmark and Kubernetes: + +| Hardening Guide Version | Rancher Version | CIS Benchmark Version | Kubernetes Version | +| ----------------------- | --------------- | --------------------- | ------------------- | +| Hardening Guide CIS v1.6 Benchmark | Rancher v2.6.3 | CIS v1.6 | Kubernetes v1.18, v1.19, v1.20 and v1.21 | + +Because Rancher and RKE install Kubernetes services as Docker containers, many of the control verification checks in the CIS Kubernetes Benchmark do not apply and will have a result of `Not Applicable`. This guide will walk through the various controls and provide updated example commands to audit compliance in Rancher created clusters. + +This document is to be used by Rancher operators, security teams, auditors and decision makers. + +For more detail about each audit, including rationales and remediations for failing tests, you can refer to the corresponding section of the CIS Kubernetes Benchmark v1.6. You can download the benchmark, after creating a free account, in [Center for Internet Security (CIS)](https://www.cisecurity.org/benchmark/kubernetes/). + +#### Testing controls methodology + +Rancher and RKE install Kubernetes services via Docker containers. Configuration is defined by arguments passed to the container at the time of initialization, not via configuration files. + +Where control audits differ from the original CIS benchmark, the audit commands specific to Rancher are provided for testing. When performing the tests, you will need access to the Docker command line on the hosts of all three RKE roles. The commands also make use of the [kubectl](https://kubernetes.io/docs/tasks/tools/) (with a valid configuration file) and [jq](https://stedolan.github.io/jq/) tools, which are required in the testing and evaluation of test results. + +> NOTE: Only `automated` tests (previously called `scored`) are covered in this guide. + +### Controls +## 1.1 Master Node Configuration Files +### 1.1.1 Ensure that the API server pod specification file permissions are set to 644 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Cluster provisioned by RKE doesn't require or maintain a configuration file for kube-apiserver. +All configuration is passed in as arguments at container run time. + +### 1.1.2 Ensure that the API server pod specification file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Cluster provisioned by RKE doesn't require or maintain a configuration file for kube-apiserver. +All configuration is passed in as arguments at container run time. + +### 1.1.3 Ensure that the controller manager pod specification file permissions are set to 644 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Cluster provisioned by RKE doesn't require or maintain a configuration file for controller-manager. +All configuration is passed in as arguments at container run time. + +### 1.1.4 Ensure that the controller manager pod specification file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Cluster provisioned by RKE doesn't require or maintain a configuration file for controller-manager. +All configuration is passed in as arguments at container run time. + +### 1.1.5 Ensure that the scheduler pod specification file permissions are set to 644 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Cluster provisioned by RKE doesn't require or maintain a configuration file for scheduler. +All configuration is passed in as arguments at container run time. + +### 1.1.6 Ensure that the scheduler pod specification file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Cluster provisioned by RKE doesn't require or maintain a configuration file for scheduler. +All configuration is passed in as arguments at container run time. + +### 1.1.7 Ensure that the etcd pod specification file permissions are set to 644 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Cluster provisioned by RKE doesn't require or maintain a configuration file for etcd. +All configuration is passed in as arguments at container run time. + +### 1.1.8 Ensure that the etcd pod specification file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Cluster provisioned by RKE doesn't require or maintain a configuration file for etcd. +All configuration is passed in as arguments at container run time. + +### 1.1.9 Ensure that the Container Network Interface file permissions are set to 644 or more restrictive (Manual) + + +**Result:** warn + +**Remediation:** +Run the below command (based on the file location on your system) on the master node. +For example, +chmod 644 + +**Audit:** + +```bash +stat -c permissions=%a +``` + +### 1.1.10 Ensure that the Container Network Interface file ownership is set to root:root (Manual) + + +**Result:** warn + +**Remediation:** +Run the below command (based on the file location on your system) on the master node. +For example, +chown root:root + +**Audit:** + +```bash +stat -c %U:%G +``` + +### 1.1.11 Ensure that the etcd data directory permissions are set to 700 or more restrictive (Automated) + + +**Result:** pass + +**Remediation:** +On the etcd server node, get the etcd data directory, passed as an argument --data-dir, +from the below command: +ps -ef | grep etcd Run the below command (based on the etcd data directory found above). For example, +chmod 700 /var/lib/etcd + +**Audit:** + +```bash +stat -c %a /node/var/lib/etcd +``` + +**Expected Result**: + +```console +'700' is equal to '700' +``` + +**Returned Value**: + +```console +700 +``` + +### 1.1.12 Ensure that the etcd data directory ownership is set to etcd:etcd (Automated) + + +**Result:** pass + +**Remediation:** +On the etcd server node, get the etcd data directory, passed as an argument --data-dir, +from the below command: +ps -ef | grep etcd +Run the below command (based on the etcd data directory found above). +For example, chown etcd:etcd /var/lib/etcd + +A system service account is required for etcd data directory ownership. +Refer to Rancher's hardening guide for more details on how to configure this ownership. + +**Audit:** + +```bash +stat -c %U:%G /node/var/lib/etcd +``` + +**Expected Result**: + +```console +'etcd:etcd' is present +``` + +**Returned Value**: + +```console +etcd:etcd +``` + +### 1.1.13 Ensure that the admin.conf file permissions are set to 644 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Cluster provisioned by RKE does not store the kubernetes default kubeconfig credentials file on the nodes. + +### 1.1.14 Ensure that the admin.conf file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Cluster provisioned by RKE does not store the kubernetes default kubeconfig credentials file on the nodes. + +### 1.1.15 Ensure that the scheduler.conf file permissions are set to 644 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Cluster provisioned by RKE doesn't require or maintain a configuration file for scheduler. +All configuration is passed in as arguments at container run time. + +### 1.1.16 Ensure that the scheduler.conf file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Cluster provisioned by RKE doesn't require or maintain a configuration file for scheduler. +All configuration is passed in as arguments at container run time. + +### 1.1.17 Ensure that the controller-manager.conf file permissions are set to 644 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Cluster provisioned by RKE doesn't require or maintain a configuration file for controller-manager. +All configuration is passed in as arguments at container run time. + +### 1.1.18 Ensure that the controller-manager.conf file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Cluster provisioned by RKE doesn't require or maintain a configuration file for controller-manager. +All configuration is passed in as arguments at container run time. + +### 1.1.19 Ensure that the Kubernetes PKI directory and file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the master node. +For example, +chown -R root:root /etc/kubernetes/pki/ + +**Audit Script:** `check_files_owner_in_dir.sh` + +```bash +#!/usr/bin/env bash + +# This script is used to ensure the owner is set to root:root for +# the given directory and all the files in it +# +# inputs: +# $1 = /full/path/to/directory +# +# outputs: +# true/false + +INPUT_DIR=$1 + +if [[ "${INPUT_DIR}" == "" ]]; then + echo "false" + exit +fi + +if [[ $(stat -c %U:%G ${INPUT_DIR}) != "root:root" ]]; then + echo "false" + exit +fi + +statInfoLines=$(stat -c "%n %U:%G" ${INPUT_DIR}/*) +while read -r statInfoLine; do + f=$(echo ${statInfoLine} | cut -d' ' -f1) + p=$(echo ${statInfoLine} | cut -d' ' -f2) + + if [[ $(basename "$f" .pem) == "kube-etcd-"* ]]; then + if [[ "$p" != "root:root" && "$p" != "etcd:etcd" ]]; then + echo "false" + exit + fi + else + if [[ "$p" != "root:root" ]]; then + echo "false" + exit + fi + fi +done <<< "${statInfoLines}" + + +echo "true" +exit + +``` + +**Audit Execution:** + +```bash +./check_files_owner_in_dir.sh /node/etc/kubernetes/ssl +``` + +**Expected Result**: + +```console +'true' is equal to 'true' +``` + +**Returned Value**: + +```console +true +``` + +### 1.1.20 Ensure that the Kubernetes PKI certificate file permissions are set to 644 or more restrictive (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the master node. +For example, +chmod -R 644 /etc/kubernetes/pki/*.crt + +**Audit Script:** `check_files_permissions.sh` + +```bash +#!/usr/bin/env bash + +# This script is used to ensure the file permissions are set to 644 or +# more restrictive for all files in a given directory or a wildcard +# selection of files +# +# inputs: +# $1 = /full/path/to/directory or /path/to/fileswithpattern +# ex: !(*key).pem +# +# $2 (optional) = permission (ex: 600) +# +# outputs: +# true/false + +# Turn on "extended glob" for use of '!' in wildcard +shopt -s extglob + +# Turn off history to avoid surprises when using '!' +set -H + +USER_INPUT=$1 + +if [[ "${USER_INPUT}" == "" ]]; then + echo "false" + exit +fi + + +if [[ -d ${USER_INPUT} ]]; then + PATTERN="${USER_INPUT}/*" +else + PATTERN="${USER_INPUT}" +fi + +PERMISSION="" +if [[ "$2" != "" ]]; then + PERMISSION=$2 +fi + +FILES_PERMISSIONS=$(stat -c %n\ %a ${PATTERN}) + +while read -r fileInfo; do + p=$(echo ${fileInfo} | cut -d' ' -f2) + + if [[ "${PERMISSION}" != "" ]]; then + if [[ "$p" != "${PERMISSION}" ]]; then + echo "false" + exit + fi + else + if [[ "$p" != "644" && "$p" != "640" && "$p" != "600" ]]; then + echo "false" + exit + fi + fi +done <<< "${FILES_PERMISSIONS}" + + +echo "true" +exit + +``` + +**Audit Execution:** + +```bash +./check_files_permissions.sh /node/etc/kubernetes/ssl/!(*key).pem +``` + +**Expected Result**: + +```console +'true' is equal to 'true' +``` + +**Returned Value**: + +```console +true +``` + +### 1.1.21 Ensure that the Kubernetes PKI key file permissions are set to 600 (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the master node. +For example, +chmod -R 600 /etc/kubernetes/ssl/*key.pem + +**Audit Script:** `check_files_permissions.sh` + +```bash +#!/usr/bin/env bash + +# This script is used to ensure the file permissions are set to 644 or +# more restrictive for all files in a given directory or a wildcard +# selection of files +# +# inputs: +# $1 = /full/path/to/directory or /path/to/fileswithpattern +# ex: !(*key).pem +# +# $2 (optional) = permission (ex: 600) +# +# outputs: +# true/false + +# Turn on "extended glob" for use of '!' in wildcard +shopt -s extglob + +# Turn off history to avoid surprises when using '!' +set -H + +USER_INPUT=$1 + +if [[ "${USER_INPUT}" == "" ]]; then + echo "false" + exit +fi + + +if [[ -d ${USER_INPUT} ]]; then + PATTERN="${USER_INPUT}/*" +else + PATTERN="${USER_INPUT}" +fi + +PERMISSION="" +if [[ "$2" != "" ]]; then + PERMISSION=$2 +fi + +FILES_PERMISSIONS=$(stat -c %n\ %a ${PATTERN}) + +while read -r fileInfo; do + p=$(echo ${fileInfo} | cut -d' ' -f2) + + if [[ "${PERMISSION}" != "" ]]; then + if [[ "$p" != "${PERMISSION}" ]]; then + echo "false" + exit + fi + else + if [[ "$p" != "644" && "$p" != "640" && "$p" != "600" ]]; then + echo "false" + exit + fi + fi +done <<< "${FILES_PERMISSIONS}" + + +echo "true" +exit + +``` + +**Audit Execution:** + +```bash +./check_files_permissions.sh /node/etc/kubernetes/ssl/*key.pem +``` + +**Expected Result**: + +```console +'true' is equal to 'true' +``` + +**Returned Value**: + +```console +true +``` + +## 1.2 API Server +### 1.2.1 Ensure that the --anonymous-auth argument is set to false (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the master node and set the below parameter. +--anonymous-auth=false + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'false' is equal to 'false' +``` + +**Returned Value**: + +```console +root 121142 121120 7 12:27 ? 00:06:27 kube-apiserver --audit-log-maxsize=100 --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --requestheader-username-headers=X-Remote-User --bind-address=0.0.0.0 --advertise-address= --requestheader-allowed-names=kube-apiserver-proxy-client --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --allow-privileged=true --requestheader-extra-headers-prefix=X-Remote-Extra- --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --service-account-lookup=true --runtime-config=policy/v1beta1/podsecuritypolicy=true --authorization-mode=Node,RBAC --audit-log-maxage=30 --profiling=false --storage-backend=etcd3 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-servers=https://:2379 --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --secure-port=6443 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --insecure-port=0 --api-audiences=unknown --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --service-account-issuer=rke --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --requestheader-group-headers=X-Remote-Group --cloud-provider= --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-node-port-range=30000-32767 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --anonymous-auth=false --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --audit-log-format=json +``` + +### 1.2.2 Ensure that the --basic-auth-file argument is not set (Automated) + + +**Result:** pass + +**Remediation:** +Follow the documentation and configure alternate mechanisms for authentication. Then, +edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the master node and remove the --basic-auth-file= parameter. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--basic-auth-file' is not present +``` + +**Returned Value**: + +```console +root 121142 121120 7 12:27 ? 00:06:27 kube-apiserver --audit-log-maxsize=100 --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --requestheader-username-headers=X-Remote-User --bind-address=0.0.0.0 --advertise-address= --requestheader-allowed-names=kube-apiserver-proxy-client --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --allow-privileged=true --requestheader-extra-headers-prefix=X-Remote-Extra- --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --service-account-lookup=true --runtime-config=policy/v1beta1/podsecuritypolicy=true --authorization-mode=Node,RBAC --audit-log-maxage=30 --profiling=false --storage-backend=etcd3 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-servers=https://:2379 --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --secure-port=6443 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --insecure-port=0 --api-audiences=unknown --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --service-account-issuer=rke --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --requestheader-group-headers=X-Remote-Group --cloud-provider= --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-node-port-range=30000-32767 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --anonymous-auth=false --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --audit-log-format=json +``` + +### 1.2.3 Ensure that the --token-auth-file parameter is not set (Automated) + + +**Result:** pass + +**Remediation:** +Follow the documentation and configure alternate mechanisms for authentication. Then, +edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the master node and remove the --token-auth-file= parameter. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--token-auth-file' is not present +``` + +**Returned Value**: + +```console +root 121142 121120 7 12:27 ? 00:06:27 kube-apiserver --audit-log-maxsize=100 --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --requestheader-username-headers=X-Remote-User --bind-address=0.0.0.0 --advertise-address= --requestheader-allowed-names=kube-apiserver-proxy-client --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --allow-privileged=true --requestheader-extra-headers-prefix=X-Remote-Extra- --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --service-account-lookup=true --runtime-config=policy/v1beta1/podsecuritypolicy=true --authorization-mode=Node,RBAC --audit-log-maxage=30 --profiling=false --storage-backend=etcd3 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-servers=https://:2379 --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --secure-port=6443 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --insecure-port=0 --api-audiences=unknown --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --service-account-issuer=rke --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --requestheader-group-headers=X-Remote-Group --cloud-provider= --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-node-port-range=30000-32767 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --anonymous-auth=false --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --audit-log-format=json +``` + +### 1.2.4 Ensure that the --kubelet-https argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the master node and remove the --kubelet-https parameter. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--kubelet-https' is not present OR '--kubelet-https' is not present +``` + +**Returned Value**: + +```console +root 121142 121120 7 12:27 ? 00:06:27 kube-apiserver --audit-log-maxsize=100 --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --requestheader-username-headers=X-Remote-User --bind-address=0.0.0.0 --advertise-address= --requestheader-allowed-names=kube-apiserver-proxy-client --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --allow-privileged=true --requestheader-extra-headers-prefix=X-Remote-Extra- --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --service-account-lookup=true --runtime-config=policy/v1beta1/podsecuritypolicy=true --authorization-mode=Node,RBAC --audit-log-maxage=30 --profiling=false --storage-backend=etcd3 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-servers=https://:2379 --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --secure-port=6443 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --insecure-port=0 --api-audiences=unknown --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --service-account-issuer=rke --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --requestheader-group-headers=X-Remote-Group --cloud-provider= --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-node-port-range=30000-32767 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --anonymous-auth=false --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --audit-log-format=json +``` + +### 1.2.5 Ensure that the --kubelet-client-certificate and --kubelet-client-key arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection between the +apiserver and kubelets. Then, edit API server pod specification file +/etc/kubernetes/manifests/kube-apiserver.yaml on the master node and set the +kubelet client certificate and key parameters as below. +--kubelet-client-certificate= +--kubelet-client-key= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--kubelet-client-certificate' is present AND '--kubelet-client-key' is present +``` + +**Returned Value**: + +```console +root 121142 121120 7 12:27 ? 00:06:27 kube-apiserver --audit-log-maxsize=100 --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --requestheader-username-headers=X-Remote-User --bind-address=0.0.0.0 --advertise-address= --requestheader-allowed-names=kube-apiserver-proxy-client --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --allow-privileged=true --requestheader-extra-headers-prefix=X-Remote-Extra- --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --service-account-lookup=true --runtime-config=policy/v1beta1/podsecuritypolicy=true --authorization-mode=Node,RBAC --audit-log-maxage=30 --profiling=false --storage-backend=etcd3 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-servers=https://:2379 --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --secure-port=6443 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --insecure-port=0 --api-audiences=unknown --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --service-account-issuer=rke --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --requestheader-group-headers=X-Remote-Group --cloud-provider= --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-node-port-range=30000-32767 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --anonymous-auth=false --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --audit-log-format=json +``` + +### 1.2.6 Ensure that the --kubelet-certificate-authority argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and setup the TLS connection between +the apiserver and kubelets. Then, edit the API server pod specification file +/etc/kubernetes/manifests/kube-apiserver.yaml on the master node and set the +--kubelet-certificate-authority parameter to the path to the cert file for the certificate authority. +--kubelet-certificate-authority= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--kubelet-certificate-authority' is present +``` + +**Returned Value**: + +```console +root 121142 121120 7 12:27 ? 00:06:27 kube-apiserver --audit-log-maxsize=100 --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --requestheader-username-headers=X-Remote-User --bind-address=0.0.0.0 --advertise-address= --requestheader-allowed-names=kube-apiserver-proxy-client --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --allow-privileged=true --requestheader-extra-headers-prefix=X-Remote-Extra- --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --service-account-lookup=true --runtime-config=policy/v1beta1/podsecuritypolicy=true --authorization-mode=Node,RBAC --audit-log-maxage=30 --profiling=false --storage-backend=etcd3 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-servers=https://:2379 --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --secure-port=6443 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --insecure-port=0 --api-audiences=unknown --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --service-account-issuer=rke --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --requestheader-group-headers=X-Remote-Group --cloud-provider= --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-node-port-range=30000-32767 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --anonymous-auth=false --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --audit-log-format=json +``` + +### 1.2.7 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the master node and set the --authorization-mode parameter to values other than AlwaysAllow. +One such example could be as below. +--authorization-mode=RBAC + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'Node,RBAC' not have 'AlwaysAllow' +``` + +**Returned Value**: + +```console +root 121142 121120 7 12:27 ? 00:06:27 kube-apiserver --audit-log-maxsize=100 --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --requestheader-username-headers=X-Remote-User --bind-address=0.0.0.0 --advertise-address= --requestheader-allowed-names=kube-apiserver-proxy-client --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --allow-privileged=true --requestheader-extra-headers-prefix=X-Remote-Extra- --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --service-account-lookup=true --runtime-config=policy/v1beta1/podsecuritypolicy=true --authorization-mode=Node,RBAC --audit-log-maxage=30 --profiling=false --storage-backend=etcd3 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-servers=https://:2379 --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --secure-port=6443 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --insecure-port=0 --api-audiences=unknown --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --service-account-issuer=rke --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --requestheader-group-headers=X-Remote-Group --cloud-provider= --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-node-port-range=30000-32767 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --anonymous-auth=false --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --audit-log-format=json +``` + +### 1.2.8 Ensure that the --authorization-mode argument includes Node (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the master node and set the --authorization-mode parameter to a value that includes Node. +--authorization-mode=Node,RBAC + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'Node,RBAC' has 'Node' +``` + +**Returned Value**: + +```console +root 121142 121120 7 12:27 ? 00:06:27 kube-apiserver --audit-log-maxsize=100 --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --requestheader-username-headers=X-Remote-User --bind-address=0.0.0.0 --advertise-address= --requestheader-allowed-names=kube-apiserver-proxy-client --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --allow-privileged=true --requestheader-extra-headers-prefix=X-Remote-Extra- --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --service-account-lookup=true --runtime-config=policy/v1beta1/podsecuritypolicy=true --authorization-mode=Node,RBAC --audit-log-maxage=30 --profiling=false --storage-backend=etcd3 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-servers=https://:2379 --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --secure-port=6443 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --insecure-port=0 --api-audiences=unknown --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --service-account-issuer=rke --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --requestheader-group-headers=X-Remote-Group --cloud-provider= --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-node-port-range=30000-32767 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --anonymous-auth=false --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --audit-log-format=json +``` + +### 1.2.9 Ensure that the --authorization-mode argument includes RBAC (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the master node and set the --authorization-mode parameter to a value that includes RBAC, +for example: +--authorization-mode=Node,RBAC + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'Node,RBAC' has 'RBAC' +``` + +**Returned Value**: + +```console +root 121142 121120 7 12:27 ? 00:06:27 kube-apiserver --audit-log-maxsize=100 --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --requestheader-username-headers=X-Remote-User --bind-address=0.0.0.0 --advertise-address= --requestheader-allowed-names=kube-apiserver-proxy-client --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --allow-privileged=true --requestheader-extra-headers-prefix=X-Remote-Extra- --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --service-account-lookup=true --runtime-config=policy/v1beta1/podsecuritypolicy=true --authorization-mode=Node,RBAC --audit-log-maxage=30 --profiling=false --storage-backend=etcd3 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-servers=https://:2379 --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --secure-port=6443 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --insecure-port=0 --api-audiences=unknown --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --service-account-issuer=rke --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --requestheader-group-headers=X-Remote-Group --cloud-provider= --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-node-port-range=30000-32767 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --anonymous-auth=false --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --audit-log-format=json +``` + +### 1.2.10 Ensure that the admission control plugin EventRateLimit is set (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set the desired limits in a configuration file. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +and set the below parameters. +--enable-admission-plugins=...,EventRateLimit,... +--admission-control-config-file= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit' has 'EventRateLimit' +``` + +**Returned Value**: + +```console +root 121142 121120 7 12:27 ? 00:06:27 kube-apiserver --audit-log-maxsize=100 --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --requestheader-username-headers=X-Remote-User --bind-address=0.0.0.0 --advertise-address= --requestheader-allowed-names=kube-apiserver-proxy-client --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --allow-privileged=true --requestheader-extra-headers-prefix=X-Remote-Extra- --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --service-account-lookup=true --runtime-config=policy/v1beta1/podsecuritypolicy=true --authorization-mode=Node,RBAC --audit-log-maxage=30 --profiling=false --storage-backend=etcd3 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-servers=https://:2379 --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --secure-port=6443 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --insecure-port=0 --api-audiences=unknown --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --service-account-issuer=rke --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --requestheader-group-headers=X-Remote-Group --cloud-provider= --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-node-port-range=30000-32767 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --anonymous-auth=false --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --audit-log-format=json +``` + +### 1.2.11 Ensure that the admission control plugin AlwaysAdmit is not set (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the master node and either remove the --enable-admission-plugins parameter, or set it to a +value that does not include AlwaysAdmit. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit' not have 'AlwaysAdmit' OR '--enable-admission-plugins' is not present +``` + +**Returned Value**: + +```console +root 121142 121120 7 12:27 ? 00:06:27 kube-apiserver --audit-log-maxsize=100 --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --requestheader-username-headers=X-Remote-User --bind-address=0.0.0.0 --advertise-address= --requestheader-allowed-names=kube-apiserver-proxy-client --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --allow-privileged=true --requestheader-extra-headers-prefix=X-Remote-Extra- --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --service-account-lookup=true --runtime-config=policy/v1beta1/podsecuritypolicy=true --authorization-mode=Node,RBAC --audit-log-maxage=30 --profiling=false --storage-backend=etcd3 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-servers=https://:2379 --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --secure-port=6443 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --insecure-port=0 --api-audiences=unknown --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --service-account-issuer=rke --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --requestheader-group-headers=X-Remote-Group --cloud-provider= --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-node-port-range=30000-32767 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --anonymous-auth=false --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --audit-log-format=json +``` + +### 1.2.12 Ensure that the admission control plugin AlwaysPullImages is set (Manual) + + +**Result:** warn + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the master node and set the --enable-admission-plugins parameter to include +AlwaysPullImages. +--enable-admission-plugins=...,AlwaysPullImages,... + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +### 1.2.13 Ensure that the admission control plugin SecurityContextDeny is set if PodSecurityPolicy is not used (Manual) + + +**Result:** warn + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the master node and set the --enable-admission-plugins parameter to include +SecurityContextDeny, unless PodSecurityPolicy is already in place. +--enable-admission-plugins=...,SecurityContextDeny,... + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +### 1.2.14 Ensure that the admission control plugin ServiceAccount is set (Automated) + + +**Result:** pass + +**Remediation:** +Follow the documentation and create ServiceAccount objects as per your environment. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the master node and ensure that the --disable-admission-plugins parameter is set to a +value that does not include ServiceAccount. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--disable-admission-plugins' is not present OR '--disable-admission-plugins' is not present +``` + +**Returned Value**: + +```console +root 121142 121120 7 12:27 ? 00:06:28 kube-apiserver --audit-log-maxsize=100 --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --requestheader-username-headers=X-Remote-User --bind-address=0.0.0.0 --advertise-address= --requestheader-allowed-names=kube-apiserver-proxy-client --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --allow-privileged=true --requestheader-extra-headers-prefix=X-Remote-Extra- --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --service-account-lookup=true --runtime-config=policy/v1beta1/podsecuritypolicy=true --authorization-mode=Node,RBAC --audit-log-maxage=30 --profiling=false --storage-backend=etcd3 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-servers=https://:2379 --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --secure-port=6443 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --insecure-port=0 --api-audiences=unknown --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --service-account-issuer=rke --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --requestheader-group-headers=X-Remote-Group --cloud-provider= --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-node-port-range=30000-32767 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --anonymous-auth=false --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --audit-log-format=json +``` + +### 1.2.15 Ensure that the admission control plugin NamespaceLifecycle is set (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the master node and set the --disable-admission-plugins parameter to +ensure it does not include NamespaceLifecycle. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--disable-admission-plugins' is not present OR '--disable-admission-plugins' is not present +``` + +**Returned Value**: + +```console +root 121142 121120 7 12:27 ? 00:06:28 kube-apiserver --audit-log-maxsize=100 --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --requestheader-username-headers=X-Remote-User --bind-address=0.0.0.0 --advertise-address= --requestheader-allowed-names=kube-apiserver-proxy-client --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --allow-privileged=true --requestheader-extra-headers-prefix=X-Remote-Extra- --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --service-account-lookup=true --runtime-config=policy/v1beta1/podsecuritypolicy=true --authorization-mode=Node,RBAC --audit-log-maxage=30 --profiling=false --storage-backend=etcd3 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-servers=https://:2379 --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --secure-port=6443 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --insecure-port=0 --api-audiences=unknown --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --service-account-issuer=rke --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --requestheader-group-headers=X-Remote-Group --cloud-provider= --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-node-port-range=30000-32767 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --anonymous-auth=false --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --audit-log-format=json +``` + +### 1.2.16 Ensure that the admission control plugin PodSecurityPolicy is set (Automated) + + +**Result:** pass + +**Remediation:** +Follow the documentation and create Pod Security Policy objects as per your environment. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the master node and set the --enable-admission-plugins parameter to a +value that includes PodSecurityPolicy: +--enable-admission-plugins=...,PodSecurityPolicy,... +Then restart the API Server. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit' has 'PodSecurityPolicy' +``` + +**Returned Value**: + +```console +root 121142 121120 7 12:27 ? 00:06:28 kube-apiserver --audit-log-maxsize=100 --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --requestheader-username-headers=X-Remote-User --bind-address=0.0.0.0 --advertise-address= --requestheader-allowed-names=kube-apiserver-proxy-client --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --allow-privileged=true --requestheader-extra-headers-prefix=X-Remote-Extra- --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --service-account-lookup=true --runtime-config=policy/v1beta1/podsecuritypolicy=true --authorization-mode=Node,RBAC --audit-log-maxage=30 --profiling=false --storage-backend=etcd3 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-servers=https://:2379 --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --secure-port=6443 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --insecure-port=0 --api-audiences=unknown --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --service-account-issuer=rke --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --requestheader-group-headers=X-Remote-Group --cloud-provider= --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-node-port-range=30000-32767 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --anonymous-auth=false --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --audit-log-format=json +``` + +### 1.2.17 Ensure that the admission control plugin NodeRestriction is set (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and configure NodeRestriction plug-in on kubelets. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the master node and set the --enable-admission-plugins parameter to a +value that includes NodeRestriction. +--enable-admission-plugins=...,NodeRestriction,... + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit' has 'NodeRestriction' +``` + +**Returned Value**: + +```console +root 121142 121120 7 12:27 ? 00:06:28 kube-apiserver --audit-log-maxsize=100 --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --requestheader-username-headers=X-Remote-User --bind-address=0.0.0.0 --advertise-address= --requestheader-allowed-names=kube-apiserver-proxy-client --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --allow-privileged=true --requestheader-extra-headers-prefix=X-Remote-Extra- --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --service-account-lookup=true --runtime-config=policy/v1beta1/podsecuritypolicy=true --authorization-mode=Node,RBAC --audit-log-maxage=30 --profiling=false --storage-backend=etcd3 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-servers=https://:2379 --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --secure-port=6443 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --insecure-port=0 --api-audiences=unknown --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --service-account-issuer=rke --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --requestheader-group-headers=X-Remote-Group --cloud-provider= --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-node-port-range=30000-32767 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --anonymous-auth=false --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --audit-log-format=json +``` + +### 1.2.18 Ensure that the --insecure-bind-address argument is not set (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the master node and remove the --insecure-bind-address parameter. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--insecure-bind-address' is not present +``` + +**Returned Value**: + +```console +root 121142 121120 7 12:27 ? 00:06:28 kube-apiserver --audit-log-maxsize=100 --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --requestheader-username-headers=X-Remote-User --bind-address=0.0.0.0 --advertise-address= --requestheader-allowed-names=kube-apiserver-proxy-client --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --allow-privileged=true --requestheader-extra-headers-prefix=X-Remote-Extra- --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --service-account-lookup=true --runtime-config=policy/v1beta1/podsecuritypolicy=true --authorization-mode=Node,RBAC --audit-log-maxage=30 --profiling=false --storage-backend=etcd3 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-servers=https://:2379 --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --secure-port=6443 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --insecure-port=0 --api-audiences=unknown --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --service-account-issuer=rke --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --requestheader-group-headers=X-Remote-Group --cloud-provider= --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-node-port-range=30000-32767 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --anonymous-auth=false --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --audit-log-format=json +``` + +### 1.2.19 Ensure that the --insecure-port argument is set to 0 (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the master node and set the below parameter. +--insecure-port=0 + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'0' is equal to '0' +``` + +**Returned Value**: + +```console +root 121142 121120 7 12:27 ? 00:06:28 kube-apiserver --audit-log-maxsize=100 --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --requestheader-username-headers=X-Remote-User --bind-address=0.0.0.0 --advertise-address= --requestheader-allowed-names=kube-apiserver-proxy-client --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --allow-privileged=true --requestheader-extra-headers-prefix=X-Remote-Extra- --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --service-account-lookup=true --runtime-config=policy/v1beta1/podsecuritypolicy=true --authorization-mode=Node,RBAC --audit-log-maxage=30 --profiling=false --storage-backend=etcd3 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-servers=https://:2379 --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --secure-port=6443 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --insecure-port=0 --api-audiences=unknown --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --service-account-issuer=rke --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --requestheader-group-headers=X-Remote-Group --cloud-provider= --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-node-port-range=30000-32767 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --anonymous-auth=false --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --audit-log-format=json +``` + +### 1.2.20 Ensure that the --secure-port argument is not set to 0 (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the master node and either remove the --secure-port parameter or +set it to a different (non-zero) desired port. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +6443 is greater than 0 OR '--secure-port' is not present +``` + +**Returned Value**: + +```console +root 121142 121120 7 12:27 ? 00:06:28 kube-apiserver --audit-log-maxsize=100 --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --requestheader-username-headers=X-Remote-User --bind-address=0.0.0.0 --advertise-address= --requestheader-allowed-names=kube-apiserver-proxy-client --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --allow-privileged=true --requestheader-extra-headers-prefix=X-Remote-Extra- --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --service-account-lookup=true --runtime-config=policy/v1beta1/podsecuritypolicy=true --authorization-mode=Node,RBAC --audit-log-maxage=30 --profiling=false --storage-backend=etcd3 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-servers=https://:2379 --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --secure-port=6443 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --insecure-port=0 --api-audiences=unknown --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --service-account-issuer=rke --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --requestheader-group-headers=X-Remote-Group --cloud-provider= --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-node-port-range=30000-32767 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --anonymous-auth=false --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --audit-log-format=json +``` + +### 1.2.21 Ensure that the --profiling argument is set to false (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the master node and set the below parameter. +--profiling=false + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'false' is equal to 'false' +``` + +**Returned Value**: + +```console +root 121142 121120 7 12:27 ? 00:06:28 kube-apiserver --audit-log-maxsize=100 --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --requestheader-username-headers=X-Remote-User --bind-address=0.0.0.0 --advertise-address= --requestheader-allowed-names=kube-apiserver-proxy-client --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --allow-privileged=true --requestheader-extra-headers-prefix=X-Remote-Extra- --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --service-account-lookup=true --runtime-config=policy/v1beta1/podsecuritypolicy=true --authorization-mode=Node,RBAC --audit-log-maxage=30 --profiling=false --storage-backend=etcd3 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-servers=https://:2379 --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --secure-port=6443 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --insecure-port=0 --api-audiences=unknown --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --service-account-issuer=rke --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --requestheader-group-headers=X-Remote-Group --cloud-provider= --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-node-port-range=30000-32767 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --anonymous-auth=false --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --audit-log-format=json +``` + +### 1.2.22 Ensure that the --audit-log-path argument is set (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the master node and set the --audit-log-path parameter to a suitable path and +file where you would like audit logs to be written, for example: +--audit-log-path=/var/log/apiserver/audit.log + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--audit-log-path' is present +``` + +**Returned Value**: + +```console +root 121142 121120 7 12:27 ? 00:06:28 kube-apiserver --audit-log-maxsize=100 --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --requestheader-username-headers=X-Remote-User --bind-address=0.0.0.0 --advertise-address= --requestheader-allowed-names=kube-apiserver-proxy-client --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --allow-privileged=true --requestheader-extra-headers-prefix=X-Remote-Extra- --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --service-account-lookup=true --runtime-config=policy/v1beta1/podsecuritypolicy=true --authorization-mode=Node,RBAC --audit-log-maxage=30 --profiling=false --storage-backend=etcd3 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-servers=https://:2379 --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --secure-port=6443 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --insecure-port=0 --api-audiences=unknown --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --service-account-issuer=rke --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --requestheader-group-headers=X-Remote-Group --cloud-provider= --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-node-port-range=30000-32767 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --anonymous-auth=false --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --audit-log-format=json +``` + +### 1.2.23 Ensure that the --audit-log-maxage argument is set to 30 or as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the master node and set the --audit-log-maxage parameter to 30 or as an appropriate number of days: +--audit-log-maxage=30 + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +30 is greater or equal to 30 +``` + +**Returned Value**: + +```console +root 121142 121120 7 12:27 ? 00:06:28 kube-apiserver --audit-log-maxsize=100 --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --requestheader-username-headers=X-Remote-User --bind-address=0.0.0.0 --advertise-address= --requestheader-allowed-names=kube-apiserver-proxy-client --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --allow-privileged=true --requestheader-extra-headers-prefix=X-Remote-Extra- --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --service-account-lookup=true --runtime-config=policy/v1beta1/podsecuritypolicy=true --authorization-mode=Node,RBAC --audit-log-maxage=30 --profiling=false --storage-backend=etcd3 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-servers=https://:2379 --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --secure-port=6443 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --insecure-port=0 --api-audiences=unknown --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --service-account-issuer=rke --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --requestheader-group-headers=X-Remote-Group --cloud-provider= --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-node-port-range=30000-32767 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --anonymous-auth=false --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --audit-log-format=json +``` + +### 1.2.24 Ensure that the --audit-log-maxbackup argument is set to 10 or as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the master node and set the --audit-log-maxbackup parameter to 10 or to an appropriate +value. +--audit-log-maxbackup=10 + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +10 is greater or equal to 10 +``` + +**Returned Value**: + +```console +root 121142 121120 7 12:27 ? 00:06:28 kube-apiserver --audit-log-maxsize=100 --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --requestheader-username-headers=X-Remote-User --bind-address=0.0.0.0 --advertise-address= --requestheader-allowed-names=kube-apiserver-proxy-client --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --allow-privileged=true --requestheader-extra-headers-prefix=X-Remote-Extra- --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --service-account-lookup=true --runtime-config=policy/v1beta1/podsecuritypolicy=true --authorization-mode=Node,RBAC --audit-log-maxage=30 --profiling=false --storage-backend=etcd3 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-servers=https://:2379 --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --secure-port=6443 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --insecure-port=0 --api-audiences=unknown --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --service-account-issuer=rke --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --requestheader-group-headers=X-Remote-Group --cloud-provider= --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-node-port-range=30000-32767 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --anonymous-auth=false --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --audit-log-format=json +``` + +### 1.2.25 Ensure that the --audit-log-maxsize argument is set to 100 or as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the master node and set the --audit-log-maxsize parameter to an appropriate size in MB. +For example, to set it as 100 MB: +--audit-log-maxsize=100 + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +100 is greater or equal to 100 +``` + +**Returned Value**: + +```console +root 121142 121120 7 12:27 ? 00:06:28 kube-apiserver --audit-log-maxsize=100 --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --requestheader-username-headers=X-Remote-User --bind-address=0.0.0.0 --advertise-address= --requestheader-allowed-names=kube-apiserver-proxy-client --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --allow-privileged=true --requestheader-extra-headers-prefix=X-Remote-Extra- --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --service-account-lookup=true --runtime-config=policy/v1beta1/podsecuritypolicy=true --authorization-mode=Node,RBAC --audit-log-maxage=30 --profiling=false --storage-backend=etcd3 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-servers=https://:2379 --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --secure-port=6443 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --insecure-port=0 --api-audiences=unknown --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --service-account-issuer=rke --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --requestheader-group-headers=X-Remote-Group --cloud-provider= --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-node-port-range=30000-32767 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --anonymous-auth=false --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --audit-log-format=json +``` + +### 1.2.26 Ensure that the --request-timeout argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +and set the below parameter as appropriate and if needed. +For example, +--request-timeout=300s + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--request-timeout' is not present OR '--request-timeout' is not present +``` + +**Returned Value**: + +```console +root 121142 121120 7 12:27 ? 00:06:28 kube-apiserver --audit-log-maxsize=100 --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --requestheader-username-headers=X-Remote-User --bind-address=0.0.0.0 --advertise-address= --requestheader-allowed-names=kube-apiserver-proxy-client --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --allow-privileged=true --requestheader-extra-headers-prefix=X-Remote-Extra- --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --service-account-lookup=true --runtime-config=policy/v1beta1/podsecuritypolicy=true --authorization-mode=Node,RBAC --audit-log-maxage=30 --profiling=false --storage-backend=etcd3 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-servers=https://:2379 --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --secure-port=6443 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --insecure-port=0 --api-audiences=unknown --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --service-account-issuer=rke --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --requestheader-group-headers=X-Remote-Group --cloud-provider= --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-node-port-range=30000-32767 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --anonymous-auth=false --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --audit-log-format=json +``` + +### 1.2.27 Ensure that the --service-account-lookup argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the master node and set the below parameter. +--service-account-lookup=true +Alternatively, you can delete the --service-account-lookup parameter from this file so +that the default takes effect. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--service-account-lookup' is not present OR 'true' is equal to 'true' +``` + +**Returned Value**: + +```console +root 121142 121120 7 12:27 ? 00:06:28 kube-apiserver --audit-log-maxsize=100 --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --requestheader-username-headers=X-Remote-User --bind-address=0.0.0.0 --advertise-address= --requestheader-allowed-names=kube-apiserver-proxy-client --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --allow-privileged=true --requestheader-extra-headers-prefix=X-Remote-Extra- --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --service-account-lookup=true --runtime-config=policy/v1beta1/podsecuritypolicy=true --authorization-mode=Node,RBAC --audit-log-maxage=30 --profiling=false --storage-backend=etcd3 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-servers=https://:2379 --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --secure-port=6443 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --insecure-port=0 --api-audiences=unknown --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --service-account-issuer=rke --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --requestheader-group-headers=X-Remote-Group --cloud-provider= --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-node-port-range=30000-32767 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --anonymous-auth=false --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --audit-log-format=json +``` + +### 1.2.28 Ensure that the --service-account-key-file argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the master node and set the --service-account-key-file parameter +to the public key file for service accounts: +--service-account-key-file= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--service-account-key-file' is present +``` + +**Returned Value**: + +```console +root 121142 121120 7 12:27 ? 00:06:28 kube-apiserver --audit-log-maxsize=100 --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --requestheader-username-headers=X-Remote-User --bind-address=0.0.0.0 --advertise-address= --requestheader-allowed-names=kube-apiserver-proxy-client --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --allow-privileged=true --requestheader-extra-headers-prefix=X-Remote-Extra- --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --service-account-lookup=true --runtime-config=policy/v1beta1/podsecuritypolicy=true --authorization-mode=Node,RBAC --audit-log-maxage=30 --profiling=false --storage-backend=etcd3 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-servers=https://:2379 --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --secure-port=6443 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --insecure-port=0 --api-audiences=unknown --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --service-account-issuer=rke --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --requestheader-group-headers=X-Remote-Group --cloud-provider= --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-node-port-range=30000-32767 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --anonymous-auth=false --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --audit-log-format=json +``` + +### 1.2.29 Ensure that the --etcd-certfile and --etcd-keyfile arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection between the apiserver and etcd. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the master node and set the etcd certificate and key file parameters. +--etcd-certfile= +--etcd-keyfile= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--etcd-certfile' is present AND '--etcd-keyfile' is present +``` + +**Returned Value**: + +```console +root 121142 121120 7 12:27 ? 00:06:28 kube-apiserver --audit-log-maxsize=100 --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --requestheader-username-headers=X-Remote-User --bind-address=0.0.0.0 --advertise-address= --requestheader-allowed-names=kube-apiserver-proxy-client --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --allow-privileged=true --requestheader-extra-headers-prefix=X-Remote-Extra- --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --service-account-lookup=true --runtime-config=policy/v1beta1/podsecuritypolicy=true --authorization-mode=Node,RBAC --audit-log-maxage=30 --profiling=false --storage-backend=etcd3 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-servers=https://:2379 --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --secure-port=6443 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --insecure-port=0 --api-audiences=unknown --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --service-account-issuer=rke --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --requestheader-group-headers=X-Remote-Group --cloud-provider= --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-node-port-range=30000-32767 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --anonymous-auth=false --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --audit-log-format=json +``` + +### 1.2.30 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection on the apiserver. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the master node and set the TLS certificate and private key file parameters. +--tls-cert-file= +--tls-private-key-file= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--tls-cert-file' is present AND '--tls-private-key-file' is present +``` + +**Returned Value**: + +```console +root 121142 121120 7 12:27 ? 00:06:28 kube-apiserver --audit-log-maxsize=100 --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --requestheader-username-headers=X-Remote-User --bind-address=0.0.0.0 --advertise-address= --requestheader-allowed-names=kube-apiserver-proxy-client --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --allow-privileged=true --requestheader-extra-headers-prefix=X-Remote-Extra- --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --service-account-lookup=true --runtime-config=policy/v1beta1/podsecuritypolicy=true --authorization-mode=Node,RBAC --audit-log-maxage=30 --profiling=false --storage-backend=etcd3 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-servers=https://:2379 --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --secure-port=6443 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --insecure-port=0 --api-audiences=unknown --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --service-account-issuer=rke --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --requestheader-group-headers=X-Remote-Group --cloud-provider= --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-node-port-range=30000-32767 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --anonymous-auth=false --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --audit-log-format=json +``` + +### 1.2.31 Ensure that the --client-ca-file argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection on the apiserver. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the master node and set the client certificate authority file. +--client-ca-file= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--client-ca-file' is present +``` + +**Returned Value**: + +```console +root 121142 121120 7 12:27 ? 00:06:28 kube-apiserver --audit-log-maxsize=100 --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --requestheader-username-headers=X-Remote-User --bind-address=0.0.0.0 --advertise-address= --requestheader-allowed-names=kube-apiserver-proxy-client --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --allow-privileged=true --requestheader-extra-headers-prefix=X-Remote-Extra- --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --service-account-lookup=true --runtime-config=policy/v1beta1/podsecuritypolicy=true --authorization-mode=Node,RBAC --audit-log-maxage=30 --profiling=false --storage-backend=etcd3 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-servers=https://:2379 --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --secure-port=6443 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --insecure-port=0 --api-audiences=unknown --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --service-account-issuer=rke --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --requestheader-group-headers=X-Remote-Group --cloud-provider= --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-node-port-range=30000-32767 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --anonymous-auth=false --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --audit-log-format=json +``` + +### 1.2.32 Ensure that the --etcd-cafile argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection between the apiserver and etcd. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the master node and set the etcd certificate authority file parameter. +--etcd-cafile= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--etcd-cafile' is present +``` + +**Returned Value**: + +```console +root 121142 121120 7 12:27 ? 00:06:28 kube-apiserver --audit-log-maxsize=100 --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --requestheader-username-headers=X-Remote-User --bind-address=0.0.0.0 --advertise-address= --requestheader-allowed-names=kube-apiserver-proxy-client --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --allow-privileged=true --requestheader-extra-headers-prefix=X-Remote-Extra- --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --service-account-lookup=true --runtime-config=policy/v1beta1/podsecuritypolicy=true --authorization-mode=Node,RBAC --audit-log-maxage=30 --profiling=false --storage-backend=etcd3 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-servers=https://:2379 --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --secure-port=6443 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --insecure-port=0 --api-audiences=unknown --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --service-account-issuer=rke --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --requestheader-group-headers=X-Remote-Group --cloud-provider= --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-node-port-range=30000-32767 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --anonymous-auth=false --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --audit-log-format=json +``` + +### 1.2.33 Ensure that the --encryption-provider-config argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and configure a EncryptionConfig file. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the master node and set the --encryption-provider-config parameter to the path of that file: --encryption-provider-config= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--encryption-provider-config' is present +``` + +**Returned Value**: + +```console +root 121142 121120 7 12:27 ? 00:06:28 kube-apiserver --audit-log-maxsize=100 --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --requestheader-username-headers=X-Remote-User --bind-address=0.0.0.0 --advertise-address= --requestheader-allowed-names=kube-apiserver-proxy-client --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --allow-privileged=true --requestheader-extra-headers-prefix=X-Remote-Extra- --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --service-account-lookup=true --runtime-config=policy/v1beta1/podsecuritypolicy=true --authorization-mode=Node,RBAC --audit-log-maxage=30 --profiling=false --storage-backend=etcd3 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-servers=https://:2379 --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --secure-port=6443 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --insecure-port=0 --api-audiences=unknown --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --service-account-issuer=rke --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --requestheader-group-headers=X-Remote-Group --cloud-provider= --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-node-port-range=30000-32767 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --anonymous-auth=false --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --audit-log-format=json +``` + +### 1.2.34 Ensure that encryption providers are appropriately configured (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and configure a EncryptionConfig file. +In this file, choose aescbc, kms or secretbox as the encryption provider. + +**Audit Script:** `check_encryption_provider_config.sh` + +```bash +#!/usr/bin/env bash + +# This script is used to check the encrption provider config is set to aesbc +# +# outputs: +# true/false + +# TODO: Figure out the file location from the kube-apiserver commandline args +ENCRYPTION_CONFIG_FILE="/node/etc/kubernetes/ssl/encryption.yaml" + +if [[ ! -f "${ENCRYPTION_CONFIG_FILE}" ]]; then + echo "false" + exit +fi + +for provider in "$@" +do + if grep "$provider" "${ENCRYPTION_CONFIG_FILE}"; then + echo "true" + exit + fi +done + +echo "false" +exit + +``` + +**Audit Execution:** + +```bash +./check_encryption_provider_config.sh aescbc +``` + +**Expected Result**: + +```console +'true' is equal to 'true' +``` + +**Returned Value**: + +```console +- aescbc: true +``` + +### 1.2.35 Ensure that the API Server only makes use of Strong Cryptographic Ciphers (Automated) + + +**Result:** warn + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the master node and set the below parameter. +--tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM +_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM +_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM +_SHA384 + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +## 1.3 Controller Manager +### 1.3.1 Ensure that the --terminated-pod-gc-threshold argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the master node and set the --terminated-pod-gc-threshold to an appropriate threshold, +for example: +--terminated-pod-gc-threshold=10 + +**Audit:** + +```bash +/bin/ps -ef | grep kube-controller-manager | grep -v grep +``` + +**Expected Result**: + +```console +'--terminated-pod-gc-threshold' is present +``` + +**Returned Value**: + +```console +root 121366 121346 1 12:27 ? 00:01:13 kube-controller-manager --cluster-cidr=10.42.0.0/16 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --allocate-node-cidrs=true --configure-cloud-routes=false --leader-elect=true --pod-eviction-timeout=5m0s --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --feature-gates=RotateKubeletServerCertificate=true --bind-address=127.0.0.1 --enable-hostpath-provisioner=false --address=127.0.0.1 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --cloud-provider= --service-account-private-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --service-cluster-ip-range=10.43.0.0/16 --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --root-ca-file=/etc/kubernetes/ssl/kube-ca.pem --node-monitor-grace-period=40s --profiling=false --terminated-pod-gc-threshold=1000 --v=2 --allow-untagged-cloud=true --use-service-account-credentials=true +``` + +### 1.3.2 Ensure that the --profiling argument is set to false (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the master node and set the below parameter. +--profiling=false + +**Audit:** + +```bash +/bin/ps -ef | grep kube-controller-manager | grep -v grep +``` + +**Expected Result**: + +```console +'false' is equal to 'false' +``` + +**Returned Value**: + +```console +root 121366 121346 1 12:27 ? 00:01:13 kube-controller-manager --cluster-cidr=10.42.0.0/16 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --allocate-node-cidrs=true --configure-cloud-routes=false --leader-elect=true --pod-eviction-timeout=5m0s --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --feature-gates=RotateKubeletServerCertificate=true --bind-address=127.0.0.1 --enable-hostpath-provisioner=false --address=127.0.0.1 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --cloud-provider= --service-account-private-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --service-cluster-ip-range=10.43.0.0/16 --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --root-ca-file=/etc/kubernetes/ssl/kube-ca.pem --node-monitor-grace-period=40s --profiling=false --terminated-pod-gc-threshold=1000 --v=2 --allow-untagged-cloud=true --use-service-account-credentials=true +``` + +### 1.3.3 Ensure that the --use-service-account-credentials argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the master node to set the below parameter. +--use-service-account-credentials=true + +**Audit:** + +```bash +/bin/ps -ef | grep kube-controller-manager | grep -v grep +``` + +**Expected Result**: + +```console +'true' is not equal to 'false' +``` + +**Returned Value**: + +```console +root 121366 121346 1 12:27 ? 00:01:13 kube-controller-manager --cluster-cidr=10.42.0.0/16 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --allocate-node-cidrs=true --configure-cloud-routes=false --leader-elect=true --pod-eviction-timeout=5m0s --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --feature-gates=RotateKubeletServerCertificate=true --bind-address=127.0.0.1 --enable-hostpath-provisioner=false --address=127.0.0.1 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --cloud-provider= --service-account-private-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --service-cluster-ip-range=10.43.0.0/16 --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --root-ca-file=/etc/kubernetes/ssl/kube-ca.pem --node-monitor-grace-period=40s --profiling=false --terminated-pod-gc-threshold=1000 --v=2 --allow-untagged-cloud=true --use-service-account-credentials=true +``` + +### 1.3.4 Ensure that the --service-account-private-key-file argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the master node and set the --service-account-private-key-file parameter +to the private key file for service accounts. +--service-account-private-key-file= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-controller-manager | grep -v grep +``` + +**Expected Result**: + +```console +'--service-account-private-key-file' is present +``` + +**Returned Value**: + +```console +root 121366 121346 1 12:27 ? 00:01:13 kube-controller-manager --cluster-cidr=10.42.0.0/16 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --allocate-node-cidrs=true --configure-cloud-routes=false --leader-elect=true --pod-eviction-timeout=5m0s --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --feature-gates=RotateKubeletServerCertificate=true --bind-address=127.0.0.1 --enable-hostpath-provisioner=false --address=127.0.0.1 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --cloud-provider= --service-account-private-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --service-cluster-ip-range=10.43.0.0/16 --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --root-ca-file=/etc/kubernetes/ssl/kube-ca.pem --node-monitor-grace-period=40s --profiling=false --terminated-pod-gc-threshold=1000 --v=2 --allow-untagged-cloud=true --use-service-account-credentials=true +``` + +### 1.3.5 Ensure that the --root-ca-file argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the master node and set the --root-ca-file parameter to the certificate bundle file`. +--root-ca-file= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-controller-manager | grep -v grep +``` + +**Expected Result**: + +```console +'--root-ca-file' is present +``` + +**Returned Value**: + +```console +root 121366 121346 1 12:27 ? 00:01:13 kube-controller-manager --cluster-cidr=10.42.0.0/16 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --allocate-node-cidrs=true --configure-cloud-routes=false --leader-elect=true --pod-eviction-timeout=5m0s --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --feature-gates=RotateKubeletServerCertificate=true --bind-address=127.0.0.1 --enable-hostpath-provisioner=false --address=127.0.0.1 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --cloud-provider= --service-account-private-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --service-cluster-ip-range=10.43.0.0/16 --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --root-ca-file=/etc/kubernetes/ssl/kube-ca.pem --node-monitor-grace-period=40s --profiling=false --terminated-pod-gc-threshold=1000 --v=2 --allow-untagged-cloud=true --use-service-account-credentials=true +``` + +### 1.3.6 Ensure that the RotateKubeletServerCertificate argument is set to true (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the master node and set the --feature-gates parameter to include RotateKubeletServerCertificate=true. +--feature-gates=RotateKubeletServerCertificate=true + +Cluster provisioned by RKE handles certificate rotation directly through RKE. + +### 1.3.7 Ensure that the --bind-address argument is set to 127.0.0.1 (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the master node and ensure the correct value for the --bind-address parameter + +**Audit:** + +```bash +/bin/ps -ef | grep kube-controller-manager | grep -v grep +``` + +**Expected Result**: + +```console +'127.0.0.1' is equal to '127.0.0.1' OR '--bind-address' is not present +``` + +**Returned Value**: + +```console +root 121366 121346 1 12:27 ? 00:01:13 kube-controller-manager --cluster-cidr=10.42.0.0/16 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --allocate-node-cidrs=true --configure-cloud-routes=false --leader-elect=true --pod-eviction-timeout=5m0s --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --feature-gates=RotateKubeletServerCertificate=true --bind-address=127.0.0.1 --enable-hostpath-provisioner=false --address=127.0.0.1 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --cloud-provider= --service-account-private-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --service-cluster-ip-range=10.43.0.0/16 --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --root-ca-file=/etc/kubernetes/ssl/kube-ca.pem --node-monitor-grace-period=40s --profiling=false --terminated-pod-gc-threshold=1000 --v=2 --allow-untagged-cloud=true --use-service-account-credentials=true +``` + +## 1.4 Scheduler +### 1.4.1 Ensure that the --profiling argument is set to false (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Scheduler pod specification file /etc/kubernetes/manifests/kube-scheduler.yaml file +on the master node and set the below parameter. +--profiling=false + +**Audit:** + +```bash +/bin/ps -ef | grep kube-scheduler | grep -v grep +``` + +**Expected Result**: + +```console +'false' is equal to 'false' +``` + +**Returned Value**: + +```console +root 121587 121567 0 12:27 ? 00:00:12 kube-scheduler --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-scheduler.yaml --address=127.0.0.1 --leader-elect=true --profiling=false --v=2 --bind-address=127.0.0.1 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 +``` + +### 1.4.2 Ensure that the --bind-address argument is set to 127.0.0.1 (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Scheduler pod specification file /etc/kubernetes/manifests/kube-scheduler.yaml +on the master node and ensure the correct value for the --bind-address parameter + +**Audit:** + +```bash +/bin/ps -ef | grep kube-scheduler | grep -v grep +``` + +**Expected Result**: + +```console +'127.0.0.1' is equal to '127.0.0.1' OR '--bind-address' is not present +``` + +**Returned Value**: + +```console +root 121587 121567 0 12:27 ? 00:00:12 kube-scheduler --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-scheduler.yaml --address=127.0.0.1 --leader-elect=true --profiling=false --v=2 --bind-address=127.0.0.1 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 +``` + +## 2 Etcd Node Configuration Files +### 2.1 Ensure that the --cert-file and --key-file arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the etcd service documentation and configure TLS encryption. +Then, edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml +on the master node and set the below parameters. +--cert-file= +--key-file= + +**Audit:** + +```bash +/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep +``` + +**Expected Result**: + +```console +'--cert-file' is present AND '--key-file' is present +``` + +**Returned Value**: + +```console +etcd 120679 120657 1 12:27 ? 00:01:17 /usr/local/bin/etcd --trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --cert-file=/etc/kubernetes/ssl/kube-etcd-.pem --heartbeat-interval=500 --election-timeout=5000 --initial-cluster-token=etcd-cluster-1 --initial-cluster=etcd-=https://:2380 --peer-trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --peer-cert-file=/etc/kubernetes/ssl/kube-etcd-.pem --peer-key-file=/etc/kubernetes/ssl/kube-etcd--key.pem --peer-client-cert-auth=true --data-dir=/var/lib/rancher/etcd/ --initial-advertise-peer-urls=https://:2380 --initial-cluster-state=new --advertise-client-urls=https://:2379 --client-cert-auth=true --enable-v2=true --name=etcd- --listen-client-urls=https://:2379 --listen-peer-urls=https://:2380 --key-file=/etc/kubernetes/ssl/kube-etcd--key.pem --cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 root 120728 120707 0 12:27 ? 00:00:00 /opt/rke-tools/rke-etcd-backup etcd-backup save --cacert /etc/kubernetes/ssl/kube-ca.pem --cert /etc/kubernetes/ssl/kube-node.pem --key /etc/kubernetes/ssl/kube-node-key.pem --name etcd-rolling-snapshots --endpoints=:2379 --retention=72h --creation=12h root 121142 121120 7 12:27 ? 00:06:27 kube-apiserver --audit-log-maxsize=100 --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --requestheader-username-headers=X-Remote-User --bind-address=0.0.0.0 --advertise-address= --requestheader-allowed-names=kube-apiserver-proxy-client --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --allow-privileged=true --requestheader-extra-headers-prefix=X-Remote-Extra- --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --service-account-lookup=true --runtime-config=policy/v1beta1/podsecuritypolicy=true --authorization-mode=Node,RBAC --audit-log-maxage=30 --profiling=false --storage-backend=etcd3 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-servers=https://:2379 --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --secure-port=6443 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --insecure-port=0 --api-audiences=unknown --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --service-account-issuer=rke --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --requestheader-group-headers=X-Remote-Group --cloud-provider= --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-node-port-range=30000-32767 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --anonymous-auth=false --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --audit-log-format=json root 214939 214868 2 13:56 ? 00:00:00 kube-bench run --targets etcd --scored --nosummary --noremediations --v=5 --config-dir=/etc/kube-bench/cfg --benchmark rke-cis-1.6-hardened --json --log_dir /tmp/results/logs --outputfile /tmp/results/etcd.json +``` + +### 2.2 Ensure that the --client-cert-auth argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the master +node and set the below parameter. +--client-cert-auth="true" + +**Audit:** + +```bash +/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep +``` + +**Expected Result**: + +```console +'--client-cert-auth' is present OR 'true' is equal to 'true' +``` + +**Returned Value**: + +```console +etcd 120679 120657 1 12:27 ? 00:01:17 /usr/local/bin/etcd --trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --cert-file=/etc/kubernetes/ssl/kube-etcd-.pem --heartbeat-interval=500 --election-timeout=5000 --initial-cluster-token=etcd-cluster-1 --initial-cluster=etcd-=https://:2380 --peer-trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --peer-cert-file=/etc/kubernetes/ssl/kube-etcd-.pem --peer-key-file=/etc/kubernetes/ssl/kube-etcd--key.pem --peer-client-cert-auth=true --data-dir=/var/lib/rancher/etcd/ --initial-advertise-peer-urls=https://:2380 --initial-cluster-state=new --advertise-client-urls=https://:2379 --client-cert-auth=true --enable-v2=true --name=etcd- --listen-client-urls=https://:2379 --listen-peer-urls=https://:2380 --key-file=/etc/kubernetes/ssl/kube-etcd--key.pem --cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 root 120728 120707 0 12:27 ? 00:00:00 /opt/rke-tools/rke-etcd-backup etcd-backup save --cacert /etc/kubernetes/ssl/kube-ca.pem --cert /etc/kubernetes/ssl/kube-node.pem --key /etc/kubernetes/ssl/kube-node-key.pem --name etcd-rolling-snapshots --endpoints=:2379 --retention=72h --creation=12h root 121142 121120 7 12:27 ? 00:06:27 kube-apiserver --audit-log-maxsize=100 --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --requestheader-username-headers=X-Remote-User --bind-address=0.0.0.0 --advertise-address= --requestheader-allowed-names=kube-apiserver-proxy-client --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --allow-privileged=true --requestheader-extra-headers-prefix=X-Remote-Extra- --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --service-account-lookup=true --runtime-config=policy/v1beta1/podsecuritypolicy=true --authorization-mode=Node,RBAC --audit-log-maxage=30 --profiling=false --storage-backend=etcd3 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-servers=https://:2379 --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --secure-port=6443 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --insecure-port=0 --api-audiences=unknown --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --service-account-issuer=rke --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --requestheader-group-headers=X-Remote-Group --cloud-provider= --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-node-port-range=30000-32767 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --anonymous-auth=false --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --audit-log-format=json root 214939 214868 2 13:56 ? 00:00:00 kube-bench run --targets etcd --scored --nosummary --noremediations --v=5 --config-dir=/etc/kube-bench/cfg --benchmark rke-cis-1.6-hardened --json --log_dir /tmp/results/logs --outputfile /tmp/results/etcd.json +``` + +### 2.3 Ensure that the --auto-tls argument is not set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the master +node and either remove the --auto-tls parameter or set it to false. + --auto-tls=false + +**Audit:** + +```bash +/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep +``` + +**Expected Result**: + +```console +'--auto-tls' is not present OR '--auto-tls' is not present +``` + +**Returned Value**: + +```console +etcd 120679 120657 1 12:27 ? 00:01:17 /usr/local/bin/etcd --trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --cert-file=/etc/kubernetes/ssl/kube-etcd-.pem --heartbeat-interval=500 --election-timeout=5000 --initial-cluster-token=etcd-cluster-1 --initial-cluster=etcd-=https://:2380 --peer-trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --peer-cert-file=/etc/kubernetes/ssl/kube-etcd-.pem --peer-key-file=/etc/kubernetes/ssl/kube-etcd--key.pem --peer-client-cert-auth=true --data-dir=/var/lib/rancher/etcd/ --initial-advertise-peer-urls=https://:2380 --initial-cluster-state=new --advertise-client-urls=https://:2379 --client-cert-auth=true --enable-v2=true --name=etcd- --listen-client-urls=https://:2379 --listen-peer-urls=https://:2380 --key-file=/etc/kubernetes/ssl/kube-etcd--key.pem --cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 root 120728 120707 0 12:27 ? 00:00:00 /opt/rke-tools/rke-etcd-backup etcd-backup save --cacert /etc/kubernetes/ssl/kube-ca.pem --cert /etc/kubernetes/ssl/kube-node.pem --key /etc/kubernetes/ssl/kube-node-key.pem --name etcd-rolling-snapshots --endpoints=:2379 --retention=72h --creation=12h root 121142 121120 7 12:27 ? 00:06:27 kube-apiserver --audit-log-maxsize=100 --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --requestheader-username-headers=X-Remote-User --bind-address=0.0.0.0 --advertise-address= --requestheader-allowed-names=kube-apiserver-proxy-client --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --allow-privileged=true --requestheader-extra-headers-prefix=X-Remote-Extra- --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --service-account-lookup=true --runtime-config=policy/v1beta1/podsecuritypolicy=true --authorization-mode=Node,RBAC --audit-log-maxage=30 --profiling=false --storage-backend=etcd3 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-servers=https://:2379 --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --secure-port=6443 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --insecure-port=0 --api-audiences=unknown --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --service-account-issuer=rke --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --requestheader-group-headers=X-Remote-Group --cloud-provider= --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-node-port-range=30000-32767 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --anonymous-auth=false --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --audit-log-format=json root 214939 214868 1 13:56 ? 00:00:00 kube-bench run --targets etcd --scored --nosummary --noremediations --v=5 --config-dir=/etc/kube-bench/cfg --benchmark rke-cis-1.6-hardened --json --log_dir /tmp/results/logs --outputfile /tmp/results/etcd.json +``` + +### 2.4 Ensure that the --peer-cert-file and --peer-key-file arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the etcd service documentation and configure peer TLS encryption as appropriate +for your etcd cluster. +Then, edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the +master node and set the below parameters. +--peer-client-file= +--peer-key-file= + +**Audit:** + +```bash +/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep +``` + +**Expected Result**: + +```console +'--peer-cert-file' is present AND '--peer-key-file' is present +``` + +**Returned Value**: + +```console +etcd 120679 120657 1 12:27 ? 00:01:17 /usr/local/bin/etcd --trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --cert-file=/etc/kubernetes/ssl/kube-etcd-.pem --heartbeat-interval=500 --election-timeout=5000 --initial-cluster-token=etcd-cluster-1 --initial-cluster=etcd-=https://:2380 --peer-trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --peer-cert-file=/etc/kubernetes/ssl/kube-etcd-.pem --peer-key-file=/etc/kubernetes/ssl/kube-etcd--key.pem --peer-client-cert-auth=true --data-dir=/var/lib/rancher/etcd/ --initial-advertise-peer-urls=https://:2380 --initial-cluster-state=new --advertise-client-urls=https://:2379 --client-cert-auth=true --enable-v2=true --name=etcd- --listen-client-urls=https://:2379 --listen-peer-urls=https://:2380 --key-file=/etc/kubernetes/ssl/kube-etcd--key.pem --cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 root 120728 120707 0 12:27 ? 00:00:00 /opt/rke-tools/rke-etcd-backup etcd-backup save --cacert /etc/kubernetes/ssl/kube-ca.pem --cert /etc/kubernetes/ssl/kube-node.pem --key /etc/kubernetes/ssl/kube-node-key.pem --name etcd-rolling-snapshots --endpoints=:2379 --retention=72h --creation=12h root 121142 121120 7 12:27 ? 00:06:27 kube-apiserver --audit-log-maxsize=100 --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --requestheader-username-headers=X-Remote-User --bind-address=0.0.0.0 --advertise-address= --requestheader-allowed-names=kube-apiserver-proxy-client --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --allow-privileged=true --requestheader-extra-headers-prefix=X-Remote-Extra- --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --service-account-lookup=true --runtime-config=policy/v1beta1/podsecuritypolicy=true --authorization-mode=Node,RBAC --audit-log-maxage=30 --profiling=false --storage-backend=etcd3 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-servers=https://:2379 --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --secure-port=6443 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --insecure-port=0 --api-audiences=unknown --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --service-account-issuer=rke --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --requestheader-group-headers=X-Remote-Group --cloud-provider= --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-node-port-range=30000-32767 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --anonymous-auth=false --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --audit-log-format=json root 214939 214868 5 13:56 ? 00:00:00 kube-bench run --targets etcd --scored --nosummary --noremediations --v=5 --config-dir=/etc/kube-bench/cfg --benchmark rke-cis-1.6-hardened --json --log_dir /tmp/results/logs --outputfile /tmp/results/etcd.json +``` + +### 2.5 Ensure that the --peer-client-cert-auth argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the master +node and set the below parameter. +--peer-client-cert-auth=true + +**Audit:** + +```bash +/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep +``` + +**Expected Result**: + +```console +'--peer-client-cert-auth' is present OR 'true' is equal to 'true' +``` + +**Returned Value**: + +```console +etcd 120679 120657 1 12:27 ? 00:01:17 /usr/local/bin/etcd --trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --cert-file=/etc/kubernetes/ssl/kube-etcd-.pem --heartbeat-interval=500 --election-timeout=5000 --initial-cluster-token=etcd-cluster-1 --initial-cluster=etcd-=https://:2380 --peer-trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --peer-cert-file=/etc/kubernetes/ssl/kube-etcd-.pem --peer-key-file=/etc/kubernetes/ssl/kube-etcd--key.pem --peer-client-cert-auth=true --data-dir=/var/lib/rancher/etcd/ --initial-advertise-peer-urls=https://:2380 --initial-cluster-state=new --advertise-client-urls=https://:2379 --client-cert-auth=true --enable-v2=true --name=etcd- --listen-client-urls=https://:2379 --listen-peer-urls=https://:2380 --key-file=/etc/kubernetes/ssl/kube-etcd--key.pem --cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 root 120728 120707 0 12:27 ? 00:00:00 /opt/rke-tools/rke-etcd-backup etcd-backup save --cacert /etc/kubernetes/ssl/kube-ca.pem --cert /etc/kubernetes/ssl/kube-node.pem --key /etc/kubernetes/ssl/kube-node-key.pem --name etcd-rolling-snapshots --endpoints=:2379 --retention=72h --creation=12h root 121142 121120 7 12:27 ? 00:06:27 kube-apiserver --audit-log-maxsize=100 --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --requestheader-username-headers=X-Remote-User --bind-address=0.0.0.0 --advertise-address= --requestheader-allowed-names=kube-apiserver-proxy-client --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --allow-privileged=true --requestheader-extra-headers-prefix=X-Remote-Extra- --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --service-account-lookup=true --runtime-config=policy/v1beta1/podsecuritypolicy=true --authorization-mode=Node,RBAC --audit-log-maxage=30 --profiling=false --storage-backend=etcd3 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-servers=https://:2379 --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --secure-port=6443 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --insecure-port=0 --api-audiences=unknown --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --service-account-issuer=rke --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --requestheader-group-headers=X-Remote-Group --cloud-provider= --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-node-port-range=30000-32767 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --anonymous-auth=false --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --audit-log-format=json root 214939 214868 4 13:56 ? 00:00:00 kube-bench run --targets etcd --scored --nosummary --noremediations --v=5 --config-dir=/etc/kube-bench/cfg --benchmark rke-cis-1.6-hardened --json --log_dir /tmp/results/logs --outputfile /tmp/results/etcd.json +``` + +### 2.6 Ensure that the --peer-auto-tls argument is not set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the master +node and either remove the --peer-auto-tls parameter or set it to false. +--peer-auto-tls=false + +**Audit:** + +```bash +/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep +``` + +**Expected Result**: + +```console +'--peer-auto-tls' is not present OR '--peer-auto-tls' is present +``` + +**Returned Value**: + +```console +etcd 120679 120657 1 12:27 ? 00:01:17 /usr/local/bin/etcd --trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --cert-file=/etc/kubernetes/ssl/kube-etcd-.pem --heartbeat-interval=500 --election-timeout=5000 --initial-cluster-token=etcd-cluster-1 --initial-cluster=etcd-=https://:2380 --peer-trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --peer-cert-file=/etc/kubernetes/ssl/kube-etcd-.pem --peer-key-file=/etc/kubernetes/ssl/kube-etcd--key.pem --peer-client-cert-auth=true --data-dir=/var/lib/rancher/etcd/ --initial-advertise-peer-urls=https://:2380 --initial-cluster-state=new --advertise-client-urls=https://:2379 --client-cert-auth=true --enable-v2=true --name=etcd- --listen-client-urls=https://:2379 --listen-peer-urls=https://:2380 --key-file=/etc/kubernetes/ssl/kube-etcd--key.pem --cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 root 120728 120707 0 12:27 ? 00:00:00 /opt/rke-tools/rke-etcd-backup etcd-backup save --cacert /etc/kubernetes/ssl/kube-ca.pem --cert /etc/kubernetes/ssl/kube-node.pem --key /etc/kubernetes/ssl/kube-node-key.pem --name etcd-rolling-snapshots --endpoints=:2379 --retention=72h --creation=12h root 121142 121120 7 12:27 ? 00:06:27 kube-apiserver --audit-log-maxsize=100 --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --requestheader-username-headers=X-Remote-User --bind-address=0.0.0.0 --advertise-address= --requestheader-allowed-names=kube-apiserver-proxy-client --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --allow-privileged=true --requestheader-extra-headers-prefix=X-Remote-Extra- --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --service-account-lookup=true --runtime-config=policy/v1beta1/podsecuritypolicy=true --authorization-mode=Node,RBAC --audit-log-maxage=30 --profiling=false --storage-backend=etcd3 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-servers=https://:2379 --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --secure-port=6443 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --insecure-port=0 --api-audiences=unknown --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --service-account-issuer=rke --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --requestheader-group-headers=X-Remote-Group --cloud-provider= --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-node-port-range=30000-32767 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --anonymous-auth=false --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --audit-log-format=json root 214939 214868 4 13:56 ? 00:00:00 kube-bench run --targets etcd --scored --nosummary --noremediations --v=5 --config-dir=/etc/kube-bench/cfg --benchmark rke-cis-1.6-hardened --json --log_dir /tmp/results/logs --outputfile /tmp/results/etcd.json +``` + +### 2.7 Ensure that a unique Certificate Authority is used for etcd (Automated) + + +**Result:** pass + +**Remediation:** +[Manual test] +Follow the etcd documentation and create a dedicated certificate authority setup for the +etcd service. +Then, edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the +master node and set the below parameter. +--trusted-ca-file= + +**Audit:** + +```bash +/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep +``` + +**Expected Result**: + +```console +'--trusted-ca-file' is present +``` + +**Returned Value**: + +```console +etcd 120679 120657 1 12:27 ? 00:01:17 /usr/local/bin/etcd --trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --cert-file=/etc/kubernetes/ssl/kube-etcd-.pem --heartbeat-interval=500 --election-timeout=5000 --initial-cluster-token=etcd-cluster-1 --initial-cluster=etcd-=https://:2380 --peer-trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --peer-cert-file=/etc/kubernetes/ssl/kube-etcd-.pem --peer-key-file=/etc/kubernetes/ssl/kube-etcd--key.pem --peer-client-cert-auth=true --data-dir=/var/lib/rancher/etcd/ --initial-advertise-peer-urls=https://:2380 --initial-cluster-state=new --advertise-client-urls=https://:2379 --client-cert-auth=true --enable-v2=true --name=etcd- --listen-client-urls=https://:2379 --listen-peer-urls=https://:2380 --key-file=/etc/kubernetes/ssl/kube-etcd--key.pem --cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 root 120728 120707 0 12:27 ? 00:00:00 /opt/rke-tools/rke-etcd-backup etcd-backup save --cacert /etc/kubernetes/ssl/kube-ca.pem --cert /etc/kubernetes/ssl/kube-node.pem --key /etc/kubernetes/ssl/kube-node-key.pem --name etcd-rolling-snapshots --endpoints=:2379 --retention=72h --creation=12h root 121142 121120 7 12:27 ? 00:06:27 kube-apiserver --audit-log-maxsize=100 --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --requestheader-username-headers=X-Remote-User --bind-address=0.0.0.0 --advertise-address= --requestheader-allowed-names=kube-apiserver-proxy-client --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --allow-privileged=true --requestheader-extra-headers-prefix=X-Remote-Extra- --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --service-account-lookup=true --runtime-config=policy/v1beta1/podsecuritypolicy=true --authorization-mode=Node,RBAC --audit-log-maxage=30 --profiling=false --storage-backend=etcd3 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-servers=https://:2379 --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --secure-port=6443 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --insecure-port=0 --api-audiences=unknown --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --service-account-issuer=rke --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --requestheader-group-headers=X-Remote-Group --cloud-provider= --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-node-port-range=30000-32767 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --anonymous-auth=false --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --audit-log-format=json root 214939 214868 3 13:56 ? 00:00:00 kube-bench run --targets etcd --scored --nosummary --noremediations --v=5 --config-dir=/etc/kube-bench/cfg --benchmark rke-cis-1.6-hardened --json --log_dir /tmp/results/logs --outputfile /tmp/results/etcd.json +``` + +## 3.1 Authentication and Authorization +### 3.1.1 Client certificate authentication should not be used for users (Manual) + + +**Result:** warn + +**Remediation:** +Alternative mechanisms provided by Kubernetes such as the use of OIDC should be +implemented in place of client certificates. + +## 3.2 Logging +### 3.2.1 Ensure that a minimal audit policy is created (Automated) + + +**Result:** pass + +**Remediation:** +Create an audit policy file for your cluster. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--audit-policy-file' is present +``` + +**Returned Value**: + +```console +root 121142 121120 7 12:27 ? 00:06:28 kube-apiserver --audit-log-maxsize=100 --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --requestheader-username-headers=X-Remote-User --bind-address=0.0.0.0 --advertise-address= --requestheader-allowed-names=kube-apiserver-proxy-client --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --allow-privileged=true --requestheader-extra-headers-prefix=X-Remote-Extra- --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --service-account-lookup=true --runtime-config=policy/v1beta1/podsecuritypolicy=true --authorization-mode=Node,RBAC --audit-log-maxage=30 --profiling=false --storage-backend=etcd3 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-servers=https://:2379 --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --secure-port=6443 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --insecure-port=0 --api-audiences=unknown --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --service-account-issuer=rke --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --requestheader-group-headers=X-Remote-Group --cloud-provider= --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-node-port-range=30000-32767 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --anonymous-auth=false --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --audit-log-format=json +``` + +### 3.2.2 Ensure that the audit policy covers key security concerns (Manual) + + +**Result:** warn + +**Remediation:** +Consider modification of the audit policy in use on the cluster to include these items, at a +minimum. + +## 4.1 Worker Node Configuration Files +### 4.1.1 Ensure that the kubelet service file permissions are set to 644 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Cluster provisioned by RKE doesn’t require or maintain a configuration file for the kubelet service. +All configuration is passed in as arguments at container run time. + +### 4.1.2 Ensure that the kubelet service file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Cluster provisioned by RKE doesn’t require or maintain a configuration file for the kubelet service. +All configuration is passed in as arguments at container run time. + +### 4.1.3 If proxy kubeconfig file exists ensure permissions are set to 644 or more restrictive (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, +chmod 644 $proykubeconfig + +**Audit:** + +```bash +/bin/sh -c 'if test -e /node/etc/kubernetes/ssl/kubecfg-kube-proxy.yaml; then stat -c %a /node/etc/kubernetes/ssl/kubecfg-kube-proxy.yaml; fi' +``` + +**Expected Result**: + +```console +'644' is present OR '640' is present OR '600' is equal to '600' OR '444' is present OR '440' is present OR '400' is present OR '000' is present +``` + +**Returned Value**: + +```console +600 +``` + +### 4.1.4 Ensure that the proxy kubeconfig file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, chown root:root /etc/kubernetes/ssl/kubecfg-kube-proxy.yaml + +**Audit:** + +```bash +/bin/sh -c 'if test -e /etc/kubernetes/ssl/kubecfg-kube-proxy.yaml; then stat -c %U:%G /etc/kubernetes/ssl/kubecfg-kube-proxy.yaml; fi' +``` + +**Expected Result**: + +```console +'root:root' is not present OR '/etc/kubernetes/ssl/kubecfg-kube-proxy.yaml' is not present +``` + +### 4.1.5 Ensure that the --kubeconfig kubelet.conf file permissions are set to 644 or more restrictive (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, +chmod 644 /etc/kubernetes/ssl/kubecfg-kube-node.yaml + +**Audit:** + +```bash +/bin/sh -c 'if test -e /etc/kubernetes/ssl/kubecfg-kube-node.yaml; then stat -c permissions=%a /etc/kubernetes/ssl/kubecfg-kube-node.yaml; fi' +``` + +**Expected Result**: + +```console +'permissions' is not present +``` + +### 4.1.6 Ensure that the --kubeconfig kubelet.conf file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, +chown root:root /etc/kubernetes/ssl/kubecfg-kube-node.yaml + +**Audit:** + +```bash +/bin/sh -c 'if test -e /node/etc/kubernetes/ssl/kubecfg-kube-node.yaml; then stat -c %U:%G /node/etc/kubernetes/ssl/kubecfg-kube-node.yaml; fi' +``` + +**Expected Result**: + +```console +'root:root' is equal to 'root:root' +``` + +**Returned Value**: + +```console +root:root +``` + +### 4.1.7 Ensure that the certificate authorities file permissions are set to 644 or more restrictive (Automated) + + +**Result:** pass + +**Remediation:** +Run the following command to modify the file permissions of the +--client-ca-file chmod 644 + +**Audit Script:** `check_cafile_permissions.sh` + +```bash +#!/usr/bin/env bash + +CAFILE=$(ps -ef | grep kubelet | grep -v apiserver | grep -- --client-ca-file= | awk -F '--client-ca-file=' '{print $2}' | awk '{print $1}') +if test -z $CAFILE; then CAFILE=$kubeletcafile; fi +if test -e $CAFILE; then stat -c permissions=%a $CAFILE; fi + +``` + +**Audit Execution:** + +```bash +./check_cafile_permissions.sh +``` + +**Expected Result**: + +```console +'permissions' is not present +``` + +### 4.1.8 Ensure that the client certificate authorities file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the following command to modify the ownership of the --client-ca-file. +chown root:root + +**Audit Script:** `check_cafile_ownership.sh` + +```bash +#!/usr/bin/env bash + +CAFILE=$(ps -ef | grep kubelet | grep -v apiserver | grep -- --client-ca-file= | awk -F '--client-ca-file=' '{print $2}' | awk '{print $1}') +if test -z $CAFILE; then CAFILE=$kubeletcafile; fi +if test -e $CAFILE; then stat -c %U:%G $CAFILE; fi + +``` + +**Audit Execution:** + +```bash +./check_cafile_ownership.sh +``` + +**Expected Result**: + +```console +'root:root' is not present +``` + +### 4.1.9 Ensure that the kubelet --config configuration file has permissions set to 644 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the following command (using the config file location identified in the Audit step) +chmod 644 /var/lib/kubelet/config.yaml + +Clusters provisioned by RKE doesn’t require or maintain a configuration file for the kubelet. +All configuration is passed in as arguments at container run time. + +### 4.1.10 Ensure that the kubelet --config configuration file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the following command (using the config file location identified in the Audit step) +chown root:root /var/lib/kubelet/config.yaml + +Clusters provisioned by RKE doesn’t require or maintain a configuration file for the kubelet. +All configuration is passed in as arguments at container run time. + +## 4.2 Kubelet +### 4.2.1 Ensure that the anonymous-auth argument is set to false (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set authentication: anonymous: enabled to +false. +If using executable arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +--anonymous-auth=false +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/kubelet/config.yaml +``` + +**Expected Result**: + +```console +'' is not present +``` + +### 4.2.2 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set authorization: mode to Webhook. If +using executable arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_AUTHZ_ARGS variable. +--authorization-mode=Webhook +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/kubelet/config.yaml +``` + +**Expected Result**: + +```console +'' is not present +``` + +### 4.2.3 Ensure that the --client-ca-file argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set authentication: x509: clientCAFile to +the location of the client CA file. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_AUTHZ_ARGS variable. +--client-ca-file= +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/kubelet/config.yaml +``` + +**Expected Result**: + +```console +'' is not present +``` + +### 4.2.4 Ensure that the --read-only-port argument is set to 0 (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set readOnlyPort to 0. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +--read-only-port=0 +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/kubelet/config.yaml +``` + +**Expected Result**: + +```console +'' is not present OR '' is not present +``` + +### 4.2.5 Ensure that the --streaming-connection-idle-timeout argument is not set to 0 (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set streamingConnectionIdleTimeout to a +value other than 0. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +--streaming-connection-idle-timeout=5m +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/kubelet/config.yaml +``` + +**Expected Result**: + +```console +'30m' is not equal to '0' OR '--streaming-connection-idle-timeout' is not present +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 121813 121792 4 12:27 ? 00:03:37 kubelet --fail-swap-on=false --resolv-conf=/etc/resolv.conf --authorization-mode=Webhook --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --v=2 --volume-plugin-dir=/var/lib/kubelet/volumeplugins --address=0.0.0.0 --make-iptables-util-chains=true --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --hostname-override= --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-.pem --network-plugin=cni --streaming-connection-idle-timeout=30m --root-dir=/var/lib/kubelet --event-qps=0 --feature-gates=RotateKubeletServerCertificate=true --protect-kernel-defaults=true --cloud-provider= --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet--key.pem --cgroups-per-qos=True --cni-bin-dir=/opt/cni/bin --cni-conf-dir=/etc/cni/net.d --pod-infra-container-image=rancher/mirrored-pause:3.5 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --anonymous-auth=false --authentication-token-webhook=true --node-ip= --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --read-only-port=0 --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf +``` + +### 4.2.6 Ensure that the --protect-kernel-defaults argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set protectKernelDefaults: true. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +--protect-kernel-defaults=true +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/kubelet/config.yaml +``` + +**Expected Result**: + +```console +'' is not present +``` + +### 4.2.7 Ensure that the --make-iptables-util-chains argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set makeIPTablesUtilChains: true. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +remove the --make-iptables-util-chains argument from the +KUBELET_SYSTEM_PODS_ARGS variable. +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/kubelet/config.yaml +``` + +**Expected Result**: + +```console +'' is not present OR '' is not present +``` + +### 4.2.8 Ensure that the --hostname-override argument is not set (Manual) + + +**Result:** Not Applicable + +**Remediation:** +Edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf +on each worker node and remove the --hostname-override argument from the +KUBELET_SYSTEM_PODS_ARGS variable. +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service + +Clusters provisioned by RKE set the --hostname-override to avoid any hostname configuration errors + +### 4.2.9 Ensure that the --event-qps argument is set to 0 or a level which ensures appropriate event capture (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set eventRecordQPS: to an appropriate level. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/kubelet/config.yaml +``` + +**Expected Result**: + +```console +'' is not present +``` + +### 4.2.10 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set tlsCertFile to the location +of the certificate file to use to identify this Kubelet, and tlsPrivateKeyFile +to the location of the corresponding private key file. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameters in KUBELET_CERTIFICATE_ARGS variable. +--tls-cert-file= +--tls-private-key-file= +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/kubelet/config.yaml +``` + +**Expected Result**: + +```console +'' is not present AND '' is not present +``` + +### 4.2.11 Ensure that the --rotate-certificates argument is not set to false (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to add the line rotateCertificates: true or +remove it altogether to use the default value. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +remove --rotate-certificates=false argument from the KUBELET_CERTIFICATE_ARGS +variable. +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/kubelet/config.yaml +``` + +**Expected Result**: + +```console +'--rotate-certificates' is not present OR '--rotate-certificates' is not present +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 121813 121792 4 12:27 ? 00:03:37 kubelet --fail-swap-on=false --resolv-conf=/etc/resolv.conf --authorization-mode=Webhook --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --v=2 --volume-plugin-dir=/var/lib/kubelet/volumeplugins --address=0.0.0.0 --make-iptables-util-chains=true --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --hostname-override= --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-.pem --network-plugin=cni --streaming-connection-idle-timeout=30m --root-dir=/var/lib/kubelet --event-qps=0 --feature-gates=RotateKubeletServerCertificate=true --protect-kernel-defaults=true --cloud-provider= --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet--key.pem --cgroups-per-qos=True --cni-bin-dir=/opt/cni/bin --cni-conf-dir=/etc/cni/net.d --pod-infra-container-image=rancher/mirrored-pause:3.5 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --anonymous-auth=false --authentication-token-webhook=true --node-ip= --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --read-only-port=0 --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf +``` + +### 4.2.12 Verify that the RotateKubeletServerCertificate argument is set to true (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf +on each worker node and set the below parameter in KUBELET_CERTIFICATE_ARGS variable. +--feature-gates=RotateKubeletServerCertificate=true +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service + +Clusters provisioned by RKE handles certificate rotation directly through RKE. + +**Audit Config:** + +```bash +/bin/cat /var/lib/kubelet/config.yaml +``` + +### 4.2.13 Ensure that the Kubelet only makes use of Strong Cryptographic Ciphers (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set TLSCipherSuites: to +TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 +or to a subset of these values. +If using executable arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the --tls-cipher-suites parameter as follows, or to a subset of these values. +--tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/kubelet/config.yaml +``` + +**Expected Result**: + +```console +'' is not present +``` + +## 5.1 RBAC and Service Accounts +### 5.1.1 Ensure that the cluster-admin role is only used where required (Manual) + + +**Result:** warn + +**Remediation:** +Identify all clusterrolebindings to the cluster-admin role. Check if they are used and +if they need this role or if they could use a role with fewer privileges. +Where possible, first bind users to a lower privileged role and then remove the +clusterrolebinding to the cluster-admin role : +kubectl delete clusterrolebinding [name] + +### 5.1.2 Minimize access to secrets (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove get, list and watch access to secret objects in the cluster. + +### 5.1.3 Minimize wildcard use in Roles and ClusterRoles (Manual) + + +**Result:** warn + +**Remediation:** +Where possible replace any use of wildcards in clusterroles and roles with specific +objects or actions. + +### 5.1.4 Minimize access to create pods (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove create access to pod objects in the cluster. + +### 5.1.5 Ensure that default service accounts are not actively used. (Automated) + + +**Result:** pass + +**Remediation:** +Create explicit service accounts wherever a Kubernetes workload requires specific access +to the Kubernetes API server. +Modify the configuration of each default service account to include this value +automountServiceAccountToken: false + +**Audit Script:** `check_for_default_sa.sh` + +```bash +#!/bin/bash + +set -eE + +handle_error() { + echo "false" +} + +trap 'handle_error' ERR + +count_sa=$(kubectl get serviceaccounts --all-namespaces -o json | jq -r '.items[] | select(.metadata.name=="default") | select((.automountServiceAccountToken == null) or (.automountServiceAccountToken == true))' | jq .metadata.namespace | wc -l) +if [[ ${count_sa} -gt 0 ]]; then + echo "false" + exit +fi + +for ns in $(kubectl get ns --no-headers -o custom-columns=":metadata.name") +do + for result in $(kubectl get clusterrolebinding,rolebinding -n $ns -o json | jq -r '.items[] | select((.subjects[].kind=="ServiceAccount" and .subjects[].name=="default") or (.subjects[].kind=="Group" and .subjects[].name=="system:serviceaccounts"))' | jq -r '"\(.roleRef.kind),\(.roleRef.name)"') + do + read kind name <<<$(IFS=","; echo $result) + resource_count=$(kubectl get $kind $name -n $ns -o json | jq -r '.rules[] | select(.resources[] != "podsecuritypolicies")' | wc -l) + if [[ ${resource_count} -gt 0 ]]; then + echo "false" + exit + fi + done +done + + +echo "true" +``` + +**Audit Execution:** + +```bash +./check_for_default_sa.sh +``` + +**Expected Result**: + +```console +'true' is equal to 'true' +``` + +**Returned Value**: + +```console +true +``` + +### 5.1.6 Ensure that Service Account Tokens are only mounted where necessary (Manual) + + +**Result:** warn + +**Remediation:** +Modify the definition of pods and service accounts which do not need to mount service +account tokens to disable it. + +## 5.2 Pod Security Policies +### 5.2.1 Minimize the admission of privileged containers (Manual) + + +**Result:** warn + +**Remediation:** +Create a PSP as described in the Kubernetes documentation, ensuring that +the .spec.privileged field is omitted or set to false. + +### 5.2.2 Minimize the admission of containers wishing to share the host process ID namespace (Automated) + + +**Result:** pass + +**Remediation:** +Create a PSP as described in the Kubernetes documentation, ensuring that the +.spec.hostPID field is omitted or set to false. + +**Audit:** + +```bash +kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.hostPID == null) or (.spec.hostPID == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' +``` + +**Expected Result**: + +```console +1 is greater than 0 +``` + +**Returned Value**: + +```console +--count=1 +``` + +### 5.2.3 Minimize the admission of containers wishing to share the host IPC namespace (Automated) + + +**Result:** pass + +**Remediation:** +Create a PSP as described in the Kubernetes documentation, ensuring that the +.spec.hostIPC field is omitted or set to false. + +**Audit:** + +```bash +kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.hostIPC == null) or (.spec.hostIPC == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' +``` + +**Expected Result**: + +```console +1 is greater than 0 +``` + +**Returned Value**: + +```console +--count=1 +``` + +### 5.2.4 Minimize the admission of containers wishing to share the host network namespace (Automated) + + +**Result:** pass + +**Remediation:** +Create a PSP as described in the Kubernetes documentation, ensuring that the +.spec.hostNetwork field is omitted or set to false. + +**Audit:** + +```bash +kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.hostNetwork == null) or (.spec.hostNetwork == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' +``` + +**Expected Result**: + +```console +1 is greater than 0 +``` + +**Returned Value**: + +```console +--count=1 +``` + +### 5.2.5 Minimize the admission of containers with allowPrivilegeEscalation (Automated) + + +**Result:** pass + +**Remediation:** +Create a PSP as described in the Kubernetes documentation, ensuring that the +.spec.allowPrivilegeEscalation field is omitted or set to false. + +**Audit:** + +```bash +kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.allowPrivilegeEscalation == null) or (.spec.allowPrivilegeEscalation == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' +``` + +**Expected Result**: + +```console +1 is greater than 0 +``` + +**Returned Value**: + +```console +--count=1 +``` + +### 5.2.6 Minimize the admission of root containers (Manual) + + +**Result:** warn + +**Remediation:** +Create a PSP as described in the Kubernetes documentation, ensuring that the +.spec.runAsUser.rule is set to either MustRunAsNonRoot or MustRunAs with the range of +UIDs not including 0. + +### 5.2.7 Minimize the admission of containers with the NET_RAW capability (Manual) + + +**Result:** warn + +**Remediation:** +Create a PSP as described in the Kubernetes documentation, ensuring that the +.spec.requiredDropCapabilities is set to include either NET_RAW or ALL. + +### 5.2.8 Minimize the admission of containers with added capabilities (Manual) + + +**Result:** warn + +**Remediation:** +Ensure that allowedCapabilities is not present in PSPs for the cluster unless +it is set to an empty array. + +### 5.2.9 Minimize the admission of containers with capabilities assigned (Manual) + + +**Result:** warn + +**Remediation:** +Review the use of capabilites in applications runnning on your cluster. Where a namespace +contains applicaions which do not require any Linux capabities to operate consider adding +a PSP which forbids the admission of containers which do not drop all capabilities. + +## 5.3 Network Policies and CNI +### 5.3.1 Ensure that the CNI in use supports Network Policies (Manual) + + +**Result:** warn + +**Remediation:** +If the CNI plugin in use does not support network policies, consideration should be given to +making use of a different plugin, or finding an alternate mechanism for restricting traffic +in the Kubernetes cluster. + +### 5.3.2 Ensure that all Namespaces have Network Policies defined (Automated) + + +**Result:** pass + +**Remediation:** +Follow the documentation and create NetworkPolicy objects as you need them. + +**Audit Script:** `check_for_network_policies.sh` + +```bash +#!/bin/bash + +set -eE + +handle_error() { + echo "false" +} + +trap 'handle_error' ERR + +for namespace in $(kubectl get namespaces --all-namespaces -o json | jq -r '.items[].metadata.name'); do + policy_count=$(kubectl get networkpolicy -n ${namespace} -o json | jq '.items | length') + if [[ ${policy_count} -eq 0 ]]; then + echo "false" + exit + fi +done + +echo "true" + +``` + +**Audit Execution:** + +```bash +./check_for_network_policies.sh +``` + +**Expected Result**: + +```console +'true' is equal to 'true' +``` + +**Returned Value**: + +```console +true +``` + +## 5.4 Secrets Management +### 5.4.1 Prefer using secrets as files over secrets as environment variables (Manual) + + +**Result:** warn + +**Remediation:** +if possible, rewrite application code to read secrets from mounted secret files, rather than +from environment variables. + +### 5.4.2 Consider external secret storage (Manual) + + +**Result:** warn + +**Remediation:** +Refer to the secrets management options offered by your cloud provider or a third-party +secrets management solution. + +## 5.5 Extensible Admission Control +### 5.5.1 Configure Image Provenance using ImagePolicyWebhook admission controller (Manual) + + +**Result:** warn + +**Remediation:** +Follow the Kubernetes documentation and setup image provenance. + +## 5.7 General Policies +### 5.7.1 Create administrative boundaries between resources using namespaces (Manual) + + +**Result:** warn + +**Remediation:** +Follow the documentation and create namespaces for objects in your deployment as you need +them. + +### 5.7.2 Ensure that the seccomp profile is set to docker/default in your pod definitions (Manual) + + +**Result:** warn + +**Remediation:** +Seccomp is an alpha feature currently. By default, all alpha features are disabled. So, you +would need to enable alpha features in the apiserver by passing "--feature- +gates=AllAlpha=true" argument. +Edit the /etc/kubernetes/apiserver file on the master node and set the KUBE_API_ARGS +parameter to "--feature-gates=AllAlpha=true" +KUBE_API_ARGS="--feature-gates=AllAlpha=true" +Based on your system, restart the kube-apiserver service. For example: +systemctl restart kube-apiserver.service +Use annotations to enable the docker/default seccomp profile in your pod definitions. An +example is as below: +apiVersion: v1 +kind: Pod +metadata: + name: trustworthy-pod + annotations: + seccomp.security.alpha.kubernetes.io/pod: docker/default +spec: + containers: + - name: trustworthy-container + image: sotrustworthy:latest + +### 5.7.3 Apply Security Context to Your Pods and Containers (Manual) + + +**Result:** warn + +**Remediation:** +Follow the Kubernetes documentation and apply security contexts to your pods. For a +suggested list of security contexts, you may refer to the CIS Security Benchmark for Docker +Containers. + +### 5.7.4 The default namespace should not be used (Automated) + + +**Result:** pass + +**Remediation:** +Ensure that namespaces are created to allow for appropriate segregation of Kubernetes +resources and that all new resources are created in a specific namespace. + +**Audit Script:** `check_for_default_ns.sh` + +```bash +#!/bin/bash + +set -eE + +handle_error() { + echo "false" +} + +trap 'handle_error' ERR + +count=$(kubectl get all -n default -o json | jq .items[] | jq -r 'select((.metadata.name!="kubernetes"))' | jq .metadata.name | wc -l) +if [[ ${count} -gt 0 ]]; then + echo "false" + exit +fi + +echo "true" + + +``` + +**Audit Execution:** + +```bash +./check_for_default_ns.sh +``` + +**Expected Result**: + +```console +'true' is equal to 'true' +``` + +**Returned Value**: + +```console +true +``` + diff --git a/content/rancher/v2.6/en/security/hardening-guides/1.6-hardening-2.6/Rancher_v2-6_CIS_v1-6_Hardening_Guide.pdf b/content/rancher/v2.6/en/security/hardening-guides/1.6-hardening-2.6/Rancher_v2-6_CIS_v1-6_Hardening_Guide.pdf new file mode 100644 index 00000000000..c61b5c4abc1 --- /dev/null +++ b/content/rancher/v2.6/en/security/hardening-guides/1.6-hardening-2.6/Rancher_v2-6_CIS_v1-6_Hardening_Guide.pdf @@ -0,0 +1,8259 @@ +%PDF-1.7 +%🖤 +1 0 obj +<< +/Type /Pages +/Kids [ 6 0 R 8 0 R 30 0 R 45 0 R 53 0 R 67 0 R 77 0 R 99 0 R 129 0 R 164 0 R 194 0 R 232 0 R 270 0 R 310 0 R 350 0 R 390 0 R 425 0 R 464 0 R 499 0 R 534 0 R 557 0 R 591 0 R ] +/Count 22 +>> +endobj +2 0 obj +<< +/Title (Hardening Guide with CIS v1.6 Benchmark) +/Creator (pandoc) +/Producer (WeasyPrint 54.2) +>> +endobj +3 0 obj +<< +/Type /Catalog +/Pages 1 0 R +/Outlines 611 0 R +/Names << +/Dests << +/Names [ (TOC) [ 8 0 R /XYZ 77.509843 756.850394 0 ] (cb1) [ 45 0 R /XYZ 84.259843 441.572079 0 ] (cb1-1) [ 45 0 R /XYZ 101.433071 439.875057 0 ] (cb1-2) [ 45 0 R /XYZ 99.933071 420.928036 0 ] (cb1-3) [ 45 0 R /XYZ 99.933071 401.981014 0 ] (cb1-4) [ 45 0 R /XYZ 99.933071 383.033993 0 ] (cb1-5) [ 45 0 R /XYZ 99.933071 364.086971 0 ] (cb10) [ 390 0 R /XYZ 84.259843 416.811778 0 ] (cb10-1) [ 390 0 R /XYZ 101.433071 415.114756 0 ] (cb10-10) [ 390 0 R /XYZ 99.933071 244.591563 0 ] (cb10-100) [ 499 0 R /XYZ 99.933071 590.755821 0 ] (cb10-101) [ 499 0 R /XYZ 99.933071 571.808799 0 ] (cb10-102) [ 499 0 R /XYZ 99.933071 552.861778 0 ] (cb10-103) [ 499 0 R /XYZ 99.933071 533.914756 0 ] (cb10-104) [ 499 0 R /XYZ 99.933071 514.967735 0 ] (cb10-105) [ 499 0 R /XYZ 99.933071 496.020714 0 ] (cb10-106) [ 499 0 R /XYZ 99.933071 477.073692 0 ] (cb10-107) [ 499 0 R /XYZ 99.933071 458.126671 0 ] (cb10-108) [ 499 0 R /XYZ 99.933071 439.179649 0 ] (cb10-109) [ 499 0 R /XYZ 99.933071 420.232628 0 ] (cb10-11) [ 390 0 R /XYZ 99.933071 225.644542 0 ] (cb10-110) [ 499 0 R /XYZ 99.933071 401.285606 0 ] (cb10-111) [ 499 0 R /XYZ 99.933071 382.338585 0 ] (cb10-112) [ 499 0 R /XYZ 99.933071 363.391563 0 ] (cb10-113) [ 499 0 R /XYZ 99.933071 344.444542 0 ] (cb10-114) [ 499 0 R /XYZ 99.933071 325.497520 0 ] (cb10-115) [ 499 0 R /XYZ 99.933071 306.550499 0 ] (cb10-116) [ 499 0 R /XYZ 99.933071 287.603477 0 ] (cb10-117) [ 499 0 R /XYZ 99.933071 173.921348 0 ] (cb10-118) [ 499 0 R /XYZ 99.933071 154.974327 0 ] (cb10-119) [ 499 0 R /XYZ 99.933071 136.027305 0 ] (cb10-12) [ 390 0 R /XYZ 99.933071 206.697520 0 ] (cb10-120) [ 499 0 R /XYZ 99.933071 117.080284 0 ] (cb10-121) [ 499 0 R /XYZ 99.933071 98.133262 0 ] (cb10-122) [ 534 0 R /XYZ 99.933071 799.173057 0 ] (cb10-123) [ 534 0 R /XYZ 99.933071 685.490928 0 ] (cb10-124) [ 534 0 R /XYZ 99.933071 666.543907 0 ] (cb10-125) [ 534 0 R /XYZ 99.933071 647.596885 0 ] (cb10-126) [ 534 0 R /XYZ 99.933071 628.649864 0 ] (cb10-127) [ 534 0 R /XYZ 99.933071 609.702842 0 ] (cb10-128) [ 534 0 R /XYZ 99.933071 590.755821 0 ] (cb10-129) [ 534 0 R /XYZ 99.933071 571.808799 0 ] (cb10-13) [ 390 0 R /XYZ 99.933071 187.750499 0 ] (cb10-14) [ 390 0 R /XYZ 99.933071 168.803477 0 ] (cb10-15) [ 390 0 R /XYZ 99.933071 149.856456 0 ] (cb10-16) [ 390 0 R /XYZ 99.933071 130.909434 0 ] (cb10-17) [ 390 0 R /XYZ 99.933071 111.962413 0 ] (cb10-18) [ 390 0 R /XYZ 99.933071 93.015391 0 ] (cb10-19) [ 425 0 R /XYZ 99.933071 799.173057 0 ] (cb10-2) [ 390 0 R /XYZ 99.933071 396.167735 0 ] (cb10-20) [ 425 0 R /XYZ 99.933071 780.226036 0 ] (cb10-21) [ 425 0 R /XYZ 99.933071 761.279014 0 ] (cb10-22) [ 425 0 R /XYZ 99.933071 742.331993 0 ] (cb10-23) [ 425 0 R /XYZ 99.933071 723.384971 0 ] (cb10-24) [ 425 0 R /XYZ 99.933071 704.437950 0 ] (cb10-25) [ 425 0 R /XYZ 99.933071 685.490928 0 ] (cb10-26) [ 425 0 R /XYZ 99.933071 666.543907 0 ] (cb10-27) [ 425 0 R /XYZ 99.933071 647.596885 0 ] (cb10-28) [ 425 0 R /XYZ 99.933071 628.649864 0 ] (cb10-29) [ 425 0 R /XYZ 99.933071 609.702842 0 ] (cb10-3) [ 390 0 R /XYZ 99.933071 377.220714 0 ] (cb10-30) [ 425 0 R /XYZ 99.933071 571.808799 0 ] (cb10-31) [ 425 0 R /XYZ 99.933071 552.861778 0 ] (cb10-32) [ 425 0 R /XYZ 99.933071 533.914756 0 ] (cb10-33) [ 425 0 R /XYZ 99.933071 514.967735 0 ] (cb10-34) [ 425 0 R /XYZ 99.933071 496.020714 0 ] (cb10-35) [ 425 0 R /XYZ 99.933071 477.073692 0 ] (cb10-36) [ 425 0 R /XYZ 99.933071 458.126671 0 ] (cb10-37) [ 425 0 R /XYZ 99.933071 439.179649 0 ] (cb10-38) [ 425 0 R /XYZ 99.933071 420.232628 0 ] (cb10-39) [ 425 0 R /XYZ 99.933071 401.285606 0 ] (cb10-4) [ 390 0 R /XYZ 99.933071 358.273692 0 ] (cb10-40) [ 425 0 R /XYZ 99.933071 382.338585 0 ] (cb10-41) [ 425 0 R /XYZ 99.933071 363.391563 0 ] (cb10-42) [ 425 0 R /XYZ 99.933071 344.444542 0 ] (cb10-43) [ 425 0 R /XYZ 99.933071 325.497520 0 ] (cb10-44) [ 425 0 R /XYZ 99.933071 306.550499 0 ] (cb10-45) [ 425 0 R /XYZ 99.933071 287.603477 0 ] (cb10-46) [ 425 0 R /XYZ 99.933071 268.656456 0 ] (cb10-47) [ 425 0 R /XYZ 99.933071 249.709434 0 ] (cb10-48) [ 425 0 R /XYZ 99.933071 230.762413 0 ] (cb10-49) [ 425 0 R /XYZ 99.933071 211.815391 0 ] (cb10-5) [ 390 0 R /XYZ 99.933071 339.326671 0 ] (cb10-50) [ 425 0 R /XYZ 99.933071 192.868370 0 ] (cb10-51) [ 425 0 R /XYZ 99.933071 173.921348 0 ] (cb10-52) [ 425 0 R /XYZ 99.933071 154.974327 0 ] (cb10-53) [ 425 0 R /XYZ 99.933071 136.027305 0 ] (cb10-54) [ 425 0 R /XYZ 99.933071 117.080284 0 ] (cb10-55) [ 425 0 R /XYZ 99.933071 98.133262 0 ] (cb10-56) [ 464 0 R /XYZ 99.933071 799.173057 0 ] (cb10-57) [ 464 0 R /XYZ 99.933071 780.226036 0 ] (cb10-58) [ 464 0 R /XYZ 99.933071 761.279014 0 ] (cb10-59) [ 464 0 R /XYZ 99.933071 742.331993 0 ] (cb10-6) [ 390 0 R /XYZ 99.933071 320.379649 0 ] (cb10-60) [ 464 0 R /XYZ 99.933071 723.384971 0 ] (cb10-61) [ 464 0 R /XYZ 99.933071 704.437950 0 ] (cb10-62) [ 464 0 R /XYZ 99.933071 685.490928 0 ] (cb10-63) [ 464 0 R /XYZ 99.933071 666.543907 0 ] (cb10-64) [ 464 0 R /XYZ 99.933071 647.596885 0 ] (cb10-65) [ 464 0 R /XYZ 99.933071 628.649864 0 ] (cb10-66) [ 464 0 R /XYZ 99.933071 609.702842 0 ] (cb10-67) [ 464 0 R /XYZ 99.933071 590.755821 0 ] (cb10-68) [ 464 0 R /XYZ 99.933071 571.808799 0 ] (cb10-69) [ 464 0 R /XYZ 99.933071 552.861778 0 ] (cb10-7) [ 390 0 R /XYZ 99.933071 301.432628 0 ] (cb10-70) [ 464 0 R /XYZ 99.933071 533.914756 0 ] (cb10-71) [ 464 0 R /XYZ 99.933071 514.967735 0 ] (cb10-72) [ 464 0 R /XYZ 99.933071 496.020714 0 ] (cb10-73) [ 464 0 R /XYZ 99.933071 477.073692 0 ] (cb10-74) [ 464 0 R /XYZ 99.933071 458.126671 0 ] (cb10-75) [ 464 0 R /XYZ 99.933071 439.179649 0 ] (cb10-76) [ 464 0 R /XYZ 99.933071 420.232628 0 ] (cb10-77) [ 464 0 R /XYZ 99.933071 401.285606 0 ] (cb10-78) [ 464 0 R /XYZ 99.933071 382.338585 0 ] (cb10-79) [ 464 0 R /XYZ 99.933071 363.391563 0 ] (cb10-8) [ 390 0 R /XYZ 99.933071 282.485606 0 ] (cb10-80) [ 464 0 R /XYZ 99.933071 344.444542 0 ] (cb10-81) [ 464 0 R /XYZ 99.933071 325.497520 0 ] (cb10-82) [ 464 0 R /XYZ 99.933071 306.550499 0 ] (cb10-83) [ 464 0 R /XYZ 99.933071 287.603477 0 ] (cb10-84) [ 464 0 R /XYZ 99.933071 268.656456 0 ] (cb10-85) [ 464 0 R /XYZ 99.933071 249.709434 0 ] (cb10-86) [ 464 0 R /XYZ 99.933071 230.762413 0 ] (cb10-87) [ 464 0 R /XYZ 99.933071 117.080284 0 ] (cb10-88) [ 464 0 R /XYZ 99.933071 98.133262 0 ] (cb10-89) [ 499 0 R /XYZ 99.933071 799.173057 0 ] (cb10-9) [ 390 0 R /XYZ 99.933071 263.538585 0 ] (cb10-90) [ 499 0 R /XYZ 99.933071 780.226036 0 ] (cb10-91) [ 499 0 R /XYZ 99.933071 761.279014 0 ] (cb10-92) [ 499 0 R /XYZ 99.933071 742.331993 0 ] (cb10-93) [ 499 0 R /XYZ 99.933071 723.384971 0 ] (cb10-94) [ 499 0 R /XYZ 99.933071 704.437950 0 ] (cb10-95) [ 499 0 R /XYZ 99.933071 685.490928 0 ] (cb10-96) [ 499 0 R /XYZ 99.933071 666.543907 0 ] (cb10-97) [ 499 0 R /XYZ 99.933071 647.596885 0 ] (cb10-98) [ 499 0 R /XYZ 99.933071 628.649864 0 ] (cb10-99) [ 499 0 R /XYZ 99.933071 609.702842 0 ] (cb11) [ 534 0 R /XYZ 84.259843 331.262799 0 ] (cb11-1) [ 534 0 R /XYZ 101.433071 329.565778 0 ] (cb11-10) [ 534 0 R /XYZ 99.933071 159.042585 0 ] (cb11-11) [ 534 0 R /XYZ 99.933071 140.095563 0 ] (cb11-12) [ 534 0 R /XYZ 99.933071 121.148542 0 ] (cb11-13) [ 534 0 R /XYZ 99.933071 102.201520 0 ] (cb11-14) [ 557 0 R /XYZ 99.933071 799.173057 0 ] (cb11-15) [ 557 0 R /XYZ 99.933071 780.226036 0 ] (cb11-16) [ 557 0 R /XYZ 99.933071 761.279014 0 ] (cb11-17) [ 557 0 R /XYZ 99.933071 742.331993 0 ] (cb11-18) [ 557 0 R /XYZ 99.933071 723.384971 0 ] (cb11-19) [ 557 0 R /XYZ 99.933071 704.437950 0 ] (cb11-2) [ 534 0 R /XYZ 99.933071 310.618756 0 ] (cb11-20) [ 557 0 R /XYZ 99.933071 685.490928 0 ] (cb11-21) [ 557 0 R /XYZ 99.933071 666.543907 0 ] (cb11-22) [ 557 0 R /XYZ 99.933071 647.596885 0 ] (cb11-23) [ 557 0 R /XYZ 99.933071 628.649864 0 ] (cb11-24) [ 557 0 R /XYZ 99.933071 609.702842 0 ] (cb11-25) [ 557 0 R /XYZ 99.933071 590.755821 0 ] (cb11-26) [ 557 0 R /XYZ 99.933071 571.808799 0 ] (cb11-3) [ 534 0 R /XYZ 99.933071 291.671735 0 ] (cb11-4) [ 534 0 R /XYZ 99.933071 272.724714 0 ] (cb11-5) [ 534 0 R /XYZ 99.933071 253.777692 0 ] (cb11-6) [ 534 0 R /XYZ 99.933071 234.830671 0 ] (cb11-7) [ 534 0 R /XYZ 99.933071 215.883649 0 ] (cb11-8) [ 534 0 R /XYZ 99.933071 196.936628 0 ] (cb11-9) [ 534 0 R /XYZ 99.933071 177.989606 0 ] (cb12) [ 557 0 R /XYZ 84.259843 446.115778 0 ] (cb12-1) [ 557 0 R /XYZ 101.433071 444.418756 0 ] (cb12-10) [ 557 0 R /XYZ 99.933071 273.895563 0 ] (cb12-11) [ 557 0 R /XYZ 99.933071 254.948542 0 ] (cb12-12) [ 557 0 R /XYZ 99.933071 236.001520 0 ] (cb12-13) [ 557 0 R /XYZ 99.933071 217.054499 0 ] (cb12-14) [ 557 0 R /XYZ 99.933071 198.107477 0 ] (cb12-15) [ 557 0 R /XYZ 99.933071 179.160456 0 ] (cb12-16) [ 557 0 R /XYZ 99.933071 160.213434 0 ] (cb12-17) [ 557 0 R /XYZ 99.933071 141.266413 0 ] (cb12-18) [ 557 0 R /XYZ 99.933071 122.319391 0 ] (cb12-19) [ 557 0 R /XYZ 99.933071 103.372370 0 ] (cb12-2) [ 557 0 R /XYZ 99.933071 425.471735 0 ] (cb12-20) [ 591 0 R /XYZ 99.933071 780.226036 0 ] (cb12-21) [ 591 0 R /XYZ 99.933071 742.331993 0 ] (cb12-22) [ 591 0 R /XYZ 99.933071 723.384971 0 ] (cb12-23) [ 591 0 R /XYZ 99.933071 704.437950 0 ] (cb12-3) [ 557 0 R /XYZ 99.933071 406.524714 0 ] (cb12-4) [ 557 0 R /XYZ 99.933071 387.577692 0 ] (cb12-5) [ 557 0 R /XYZ 99.933071 368.630671 0 ] (cb12-6) [ 557 0 R /XYZ 99.933071 349.683649 0 ] (cb12-7) [ 557 0 R /XYZ 99.933071 330.736628 0 ] (cb12-8) [ 557 0 R /XYZ 99.933071 311.789606 0 ] (cb12-9) [ 557 0 R /XYZ 99.933071 292.842585 0 ] (cb2) [ 53 0 R /XYZ 84.259843 753.620079 0 ] (cb2-1) [ 53 0 R /XYZ 101.433071 751.923057 0 ] (cb2-2) [ 53 0 R /XYZ 99.933071 732.976036 0 ] (cb3) [ 53 0 R /XYZ 84.259843 654.779014 0 ] (cb3-1) [ 53 0 R /XYZ 101.433071 653.081993 0 ] (cb3-2) [ 53 0 R /XYZ 99.933071 634.134971 0 ] (cb3-3) [ 53 0 R /XYZ 99.933071 615.187950 0 ] (cb3-4) [ 53 0 R /XYZ 99.933071 596.240928 0 ] (cb4) [ 53 0 R /XYZ 84.259843 282.344928 0 ] (cb4-1) [ 53 0 R /XYZ 101.433071 280.647907 0 ] (cb5) [ 53 0 R /XYZ 84.259843 217.647907 0 ] (cb5-1) [ 53 0 R /XYZ 101.433071 215.950885 0 ] (cb5-2) [ 53 0 R /XYZ 99.933071 197.003864 0 ] (cb5-3) [ 53 0 R /XYZ 99.933071 178.056842 0 ] (cb5-4) [ 53 0 R /XYZ 99.933071 159.109821 0 ] (cb5-5) [ 53 0 R /XYZ 99.933071 140.162799 0 ] (cb6) [ 67 0 R /XYZ 84.259843 749.870079 0 ] (cb6-1) [ 67 0 R /XYZ 101.433071 748.173057 0 ] (cb6-2) [ 67 0 R /XYZ 99.933071 729.226036 0 ] (cb6-3) [ 67 0 R /XYZ 99.933071 710.279014 0 ] (cb6-4) [ 67 0 R /XYZ 99.933071 672.384971 0 ] (cb6-5) [ 67 0 R /XYZ 99.933071 634.490928 0 ] (cb7) [ 77 0 R /XYZ 84.259843 716.781496 0 ] (cb7-1) [ 77 0 R /XYZ 101.433071 715.084475 0 ] (cb7-10) [ 77 0 R /XYZ 99.933071 544.561281 0 ] (cb7-11) [ 77 0 R /XYZ 99.933071 525.614260 0 ] (cb7-12) [ 77 0 R /XYZ 99.933071 506.667238 0 ] (cb7-13) [ 77 0 R /XYZ 99.933071 487.720217 0 ] (cb7-14) [ 77 0 R /XYZ 99.933071 468.773195 0 ] (cb7-2) [ 77 0 R /XYZ 99.933071 696.137453 0 ] (cb7-3) [ 77 0 R /XYZ 99.933071 677.190432 0 ] (cb7-4) [ 77 0 R /XYZ 99.933071 658.243410 0 ] (cb7-5) [ 77 0 R /XYZ 99.933071 639.296389 0 ] (cb7-6) [ 77 0 R /XYZ 99.933071 620.349367 0 ] (cb7-7) [ 77 0 R /XYZ 99.933071 601.402346 0 ] (cb7-8) [ 77 0 R /XYZ 99.933071 582.455324 0 ] (cb7-9) [ 77 0 R /XYZ 99.933071 563.508303 0 ] (cb8) [ 77 0 R /XYZ 84.259843 372.023195 0 ] (cb8-1) [ 77 0 R /XYZ 101.433071 370.326174 0 ] (cb8-2) [ 77 0 R /XYZ 99.933071 351.379152 0 ] (cb8-3) [ 77 0 R /XYZ 99.933071 332.432131 0 ] (cb8-4) [ 77 0 R /XYZ 99.933071 294.538088 0 ] (cb8-5) [ 77 0 R /XYZ 99.933071 275.591066 0 ] (cb9) [ 99 0 R /XYZ 84.259843 652.281496 0 ] (cb9-1) [ 99 0 R /XYZ 101.433071 650.584475 0 ] (cb9-10) [ 99 0 R /XYZ 99.933071 423.220217 0 ] (cb9-100) [ 194 0 R /XYZ 99.933071 590.755821 0 ] (cb9-101) [ 194 0 R /XYZ 99.933071 571.808799 0 ] (cb9-102) [ 194 0 R /XYZ 99.933071 552.861778 0 ] (cb9-103) [ 194 0 R /XYZ 99.933071 533.914756 0 ] (cb9-104) [ 194 0 R /XYZ 99.933071 514.967735 0 ] (cb9-105) [ 194 0 R /XYZ 99.933071 496.020714 0 ] (cb9-106) [ 194 0 R /XYZ 99.933071 477.073692 0 ] (cb9-107) [ 194 0 R /XYZ 99.933071 458.126671 0 ] (cb9-108) [ 194 0 R /XYZ 99.933071 439.179649 0 ] (cb9-109) [ 194 0 R /XYZ 99.933071 420.232628 0 ] (cb9-11) [ 99 0 R /XYZ 99.933071 404.273195 0 ] (cb9-110) [ 194 0 R /XYZ 99.933071 401.285606 0 ] (cb9-111) [ 194 0 R /XYZ 99.933071 344.444542 0 ] (cb9-112) [ 194 0 R /XYZ 99.933071 325.497520 0 ] (cb9-113) [ 194 0 R /XYZ 99.933071 306.550499 0 ] (cb9-114) [ 194 0 R /XYZ 99.933071 287.603477 0 ] (cb9-115) [ 194 0 R /XYZ 99.933071 268.656456 0 ] (cb9-116) [ 194 0 R /XYZ 99.933071 249.709434 0 ] (cb9-117) [ 194 0 R /XYZ 99.933071 230.762413 0 ] (cb9-118) [ 194 0 R /XYZ 99.933071 211.815391 0 ] (cb9-119) [ 194 0 R /XYZ 99.933071 192.868370 0 ] (cb9-12) [ 99 0 R /XYZ 99.933071 385.326174 0 ] (cb9-120) [ 194 0 R /XYZ 99.933071 173.921348 0 ] (cb9-121) [ 194 0 R /XYZ 99.933071 154.974327 0 ] (cb9-122) [ 194 0 R /XYZ 99.933071 136.027305 0 ] (cb9-123) [ 194 0 R /XYZ 99.933071 117.080284 0 ] (cb9-124) [ 194 0 R /XYZ 99.933071 98.133262 0 ] (cb9-125) [ 232 0 R /XYZ 99.933071 799.173057 0 ] (cb9-126) [ 232 0 R /XYZ 99.933071 780.226036 0 ] (cb9-127) [ 232 0 R /XYZ 99.933071 761.279014 0 ] (cb9-128) [ 232 0 R /XYZ 99.933071 742.331993 0 ] (cb9-129) [ 232 0 R /XYZ 99.933071 704.437950 0 ] (cb9-13) [ 99 0 R /XYZ 99.933071 366.379152 0 ] (cb9-130) [ 232 0 R /XYZ 99.933071 685.490928 0 ] (cb9-131) [ 232 0 R /XYZ 99.933071 666.543907 0 ] (cb9-132) [ 232 0 R /XYZ 99.933071 647.596885 0 ] (cb9-133) [ 232 0 R /XYZ 99.933071 628.649864 0 ] (cb9-134) [ 232 0 R /XYZ 99.933071 609.702842 0 ] (cb9-135) [ 232 0 R /XYZ 99.933071 590.755821 0 ] (cb9-136) [ 232 0 R /XYZ 99.933071 571.808799 0 ] (cb9-137) [ 232 0 R /XYZ 99.933071 552.861778 0 ] (cb9-138) [ 232 0 R /XYZ 99.933071 533.914756 0 ] (cb9-139) [ 232 0 R /XYZ 99.933071 514.967735 0 ] (cb9-14) [ 99 0 R /XYZ 99.933071 347.432131 0 ] (cb9-140) [ 232 0 R /XYZ 99.933071 477.073692 0 ] (cb9-141) [ 232 0 R /XYZ 99.933071 458.126671 0 ] (cb9-142) [ 232 0 R /XYZ 99.933071 439.179649 0 ] (cb9-143) [ 232 0 R /XYZ 99.933071 420.232628 0 ] (cb9-144) [ 232 0 R /XYZ 99.933071 401.285606 0 ] (cb9-145) [ 232 0 R /XYZ 99.933071 382.338585 0 ] (cb9-146) [ 232 0 R /XYZ 99.933071 363.391563 0 ] (cb9-147) [ 232 0 R /XYZ 99.933071 344.444542 0 ] (cb9-148) [ 232 0 R /XYZ 99.933071 325.497520 0 ] (cb9-149) [ 232 0 R /XYZ 99.933071 306.550499 0 ] (cb9-15) [ 99 0 R /XYZ 99.933071 328.485109 0 ] (cb9-150) [ 232 0 R /XYZ 99.933071 287.603477 0 ] (cb9-151) [ 232 0 R /XYZ 99.933071 268.656456 0 ] (cb9-152) [ 232 0 R /XYZ 99.933071 249.709434 0 ] (cb9-153) [ 232 0 R /XYZ 99.933071 230.762413 0 ] (cb9-154) [ 232 0 R /XYZ 99.933071 211.815391 0 ] (cb9-155) [ 232 0 R /XYZ 99.933071 192.868370 0 ] (cb9-156) [ 232 0 R /XYZ 99.933071 173.921348 0 ] (cb9-157) [ 232 0 R /XYZ 99.933071 154.974327 0 ] (cb9-158) [ 232 0 R /XYZ 99.933071 136.027305 0 ] (cb9-159) [ 232 0 R /XYZ 99.933071 117.080284 0 ] (cb9-16) [ 99 0 R /XYZ 99.933071 309.538088 0 ] (cb9-160) [ 232 0 R /XYZ 99.933071 98.133262 0 ] (cb9-161) [ 270 0 R /XYZ 99.933071 799.173057 0 ] (cb9-162) [ 270 0 R /XYZ 99.933071 780.226036 0 ] (cb9-163) [ 270 0 R /XYZ 99.933071 761.279014 0 ] (cb9-164) [ 270 0 R /XYZ 99.933071 742.331993 0 ] (cb9-165) [ 270 0 R /XYZ 99.933071 723.384971 0 ] (cb9-166) [ 270 0 R /XYZ 99.933071 704.437950 0 ] (cb9-167) [ 270 0 R /XYZ 99.933071 685.490928 0 ] (cb9-168) [ 270 0 R /XYZ 99.933071 666.543907 0 ] (cb9-169) [ 270 0 R /XYZ 99.933071 647.596885 0 ] (cb9-17) [ 99 0 R /XYZ 99.933071 290.591066 0 ] (cb9-170) [ 270 0 R /XYZ 99.933071 628.649864 0 ] (cb9-171) [ 270 0 R /XYZ 99.933071 609.702842 0 ] (cb9-172) [ 270 0 R /XYZ 99.933071 590.755821 0 ] (cb9-173) [ 270 0 R /XYZ 99.933071 571.808799 0 ] (cb9-174) [ 270 0 R /XYZ 99.933071 552.861778 0 ] (cb9-175) [ 270 0 R /XYZ 99.933071 533.914756 0 ] (cb9-176) [ 270 0 R /XYZ 99.933071 514.967735 0 ] (cb9-177) [ 270 0 R /XYZ 99.933071 496.020714 0 ] (cb9-178) [ 270 0 R /XYZ 99.933071 477.073692 0 ] (cb9-179) [ 270 0 R /XYZ 99.933071 458.126671 0 ] (cb9-18) [ 99 0 R /XYZ 99.933071 271.644045 0 ] (cb9-180) [ 270 0 R /XYZ 99.933071 439.179649 0 ] (cb9-181) [ 270 0 R /XYZ 99.933071 420.232628 0 ] (cb9-182) [ 270 0 R /XYZ 99.933071 401.285606 0 ] (cb9-183) [ 270 0 R /XYZ 99.933071 382.338585 0 ] (cb9-184) [ 270 0 R /XYZ 99.933071 363.391563 0 ] (cb9-185) [ 270 0 R /XYZ 99.933071 344.444542 0 ] (cb9-186) [ 270 0 R /XYZ 99.933071 325.497520 0 ] (cb9-187) [ 270 0 R /XYZ 99.933071 306.550499 0 ] (cb9-188) [ 270 0 R /XYZ 99.933071 287.603477 0 ] (cb9-189) [ 270 0 R /XYZ 99.933071 268.656456 0 ] (cb9-19) [ 99 0 R /XYZ 99.933071 252.697023 0 ] (cb9-190) [ 270 0 R /XYZ 99.933071 249.709434 0 ] (cb9-191) [ 270 0 R /XYZ 99.933071 230.762413 0 ] (cb9-192) [ 270 0 R /XYZ 99.933071 211.815391 0 ] (cb9-193) [ 270 0 R /XYZ 99.933071 192.868370 0 ] (cb9-194) [ 270 0 R /XYZ 99.933071 173.921348 0 ] (cb9-195) [ 270 0 R /XYZ 99.933071 154.974327 0 ] (cb9-196) [ 270 0 R /XYZ 99.933071 136.027305 0 ] (cb9-197) [ 270 0 R /XYZ 99.933071 117.080284 0 ] (cb9-198) [ 270 0 R /XYZ 99.933071 98.133262 0 ] (cb9-199) [ 310 0 R /XYZ 99.933071 799.173057 0 ] (cb9-2) [ 99 0 R /XYZ 99.933071 612.690432 0 ] (cb9-20) [ 99 0 R /XYZ 99.933071 233.750002 0 ] (cb9-200) [ 310 0 R /XYZ 99.933071 780.226036 0 ] (cb9-201) [ 310 0 R /XYZ 99.933071 761.279014 0 ] (cb9-202) [ 310 0 R /XYZ 99.933071 742.331993 0 ] (cb9-203) [ 310 0 R /XYZ 99.933071 723.384971 0 ] (cb9-204) [ 310 0 R /XYZ 99.933071 704.437950 0 ] (cb9-205) [ 310 0 R /XYZ 99.933071 685.490928 0 ] (cb9-206) [ 310 0 R /XYZ 99.933071 666.543907 0 ] (cb9-207) [ 310 0 R /XYZ 99.933071 647.596885 0 ] (cb9-208) [ 310 0 R /XYZ 99.933071 628.649864 0 ] (cb9-209) [ 310 0 R /XYZ 99.933071 609.702842 0 ] (cb9-21) [ 99 0 R /XYZ 99.933071 214.802980 0 ] (cb9-210) [ 310 0 R /XYZ 99.933071 590.755821 0 ] (cb9-211) [ 310 0 R /XYZ 99.933071 571.808799 0 ] (cb9-212) [ 310 0 R /XYZ 99.933071 552.861778 0 ] (cb9-213) [ 310 0 R /XYZ 99.933071 533.914756 0 ] (cb9-214) [ 310 0 R /XYZ 99.933071 514.967735 0 ] (cb9-215) [ 310 0 R /XYZ 99.933071 496.020714 0 ] (cb9-216) [ 310 0 R /XYZ 99.933071 477.073692 0 ] (cb9-217) [ 310 0 R /XYZ 99.933071 458.126671 0 ] (cb9-218) [ 310 0 R /XYZ 99.933071 439.179649 0 ] (cb9-219) [ 310 0 R /XYZ 99.933071 420.232628 0 ] (cb9-22) [ 99 0 R /XYZ 99.933071 195.855959 0 ] (cb9-220) [ 310 0 R /XYZ 99.933071 401.285606 0 ] (cb9-221) [ 310 0 R /XYZ 99.933071 382.338585 0 ] (cb9-222) [ 310 0 R /XYZ 99.933071 363.391563 0 ] (cb9-223) [ 310 0 R /XYZ 99.933071 344.444542 0 ] (cb9-224) [ 310 0 R /XYZ 99.933071 325.497520 0 ] (cb9-225) [ 310 0 R /XYZ 99.933071 306.550499 0 ] (cb9-226) [ 310 0 R /XYZ 99.933071 287.603477 0 ] (cb9-227) [ 310 0 R /XYZ 99.933071 268.656456 0 ] (cb9-228) [ 310 0 R /XYZ 99.933071 249.709434 0 ] (cb9-229) [ 310 0 R /XYZ 99.933071 230.762413 0 ] (cb9-23) [ 99 0 R /XYZ 99.933071 176.908937 0 ] (cb9-230) [ 310 0 R /XYZ 99.933071 211.815391 0 ] (cb9-231) [ 310 0 R /XYZ 99.933071 192.868370 0 ] (cb9-232) [ 310 0 R /XYZ 99.933071 173.921348 0 ] (cb9-233) [ 310 0 R /XYZ 99.933071 154.974327 0 ] (cb9-234) [ 310 0 R /XYZ 99.933071 136.027305 0 ] (cb9-235) [ 310 0 R /XYZ 99.933071 117.080284 0 ] (cb9-236) [ 310 0 R /XYZ 99.933071 98.133262 0 ] (cb9-237) [ 350 0 R /XYZ 99.933071 799.173057 0 ] (cb9-238) [ 350 0 R /XYZ 99.933071 780.226036 0 ] (cb9-239) [ 350 0 R /XYZ 99.933071 761.279014 0 ] (cb9-24) [ 99 0 R /XYZ 99.933071 157.961916 0 ] (cb9-240) [ 350 0 R /XYZ 99.933071 742.331993 0 ] (cb9-241) [ 350 0 R /XYZ 99.933071 723.384971 0 ] (cb9-242) [ 350 0 R /XYZ 99.933071 704.437950 0 ] (cb9-243) [ 350 0 R /XYZ 99.933071 685.490928 0 ] (cb9-244) [ 350 0 R /XYZ 99.933071 666.543907 0 ] (cb9-245) [ 350 0 R /XYZ 99.933071 647.596885 0 ] (cb9-246) [ 350 0 R /XYZ 99.933071 628.649864 0 ] (cb9-247) [ 350 0 R /XYZ 99.933071 609.702842 0 ] (cb9-248) [ 350 0 R /XYZ 99.933071 590.755821 0 ] (cb9-249) [ 350 0 R /XYZ 99.933071 571.808799 0 ] (cb9-25) [ 99 0 R /XYZ 99.933071 139.014895 0 ] (cb9-250) [ 350 0 R /XYZ 99.933071 552.861778 0 ] (cb9-251) [ 350 0 R /XYZ 99.933071 533.914756 0 ] (cb9-252) [ 350 0 R /XYZ 99.933071 514.967735 0 ] (cb9-253) [ 350 0 R /XYZ 99.933071 496.020714 0 ] (cb9-254) [ 350 0 R /XYZ 99.933071 477.073692 0 ] (cb9-255) [ 350 0 R /XYZ 99.933071 458.126671 0 ] (cb9-256) [ 350 0 R /XYZ 99.933071 439.179649 0 ] (cb9-257) [ 350 0 R /XYZ 99.933071 420.232628 0 ] (cb9-258) [ 350 0 R /XYZ 99.933071 401.285606 0 ] (cb9-259) [ 350 0 R /XYZ 99.933071 382.338585 0 ] (cb9-26) [ 99 0 R /XYZ 99.933071 120.067873 0 ] (cb9-260) [ 350 0 R /XYZ 99.933071 363.391563 0 ] (cb9-261) [ 350 0 R /XYZ 99.933071 344.444542 0 ] (cb9-262) [ 350 0 R /XYZ 99.933071 325.497520 0 ] (cb9-263) [ 350 0 R /XYZ 99.933071 306.550499 0 ] (cb9-264) [ 350 0 R /XYZ 99.933071 287.603477 0 ] (cb9-265) [ 350 0 R /XYZ 99.933071 268.656456 0 ] (cb9-266) [ 350 0 R /XYZ 99.933071 249.709434 0 ] (cb9-267) [ 350 0 R /XYZ 99.933071 230.762413 0 ] (cb9-268) [ 350 0 R /XYZ 99.933071 211.815391 0 ] (cb9-269) [ 350 0 R /XYZ 99.933071 192.868370 0 ] (cb9-27) [ 99 0 R /XYZ 99.933071 101.120852 0 ] (cb9-270) [ 350 0 R /XYZ 99.933071 173.921348 0 ] (cb9-271) [ 350 0 R /XYZ 99.933071 154.974327 0 ] (cb9-272) [ 350 0 R /XYZ 99.933071 136.027305 0 ] (cb9-273) [ 350 0 R /XYZ 99.933071 117.080284 0 ] (cb9-274) [ 350 0 R /XYZ 99.933071 98.133262 0 ] (cb9-275) [ 390 0 R /XYZ 99.933071 799.173057 0 ] (cb9-276) [ 390 0 R /XYZ 99.933071 780.226036 0 ] (cb9-277) [ 390 0 R /XYZ 99.933071 761.279014 0 ] (cb9-278) [ 390 0 R /XYZ 99.933071 742.331993 0 ] (cb9-279) [ 390 0 R /XYZ 99.933071 723.384971 0 ] (cb9-28) [ 129 0 R /XYZ 99.933071 799.173057 0 ] (cb9-280) [ 390 0 R /XYZ 99.933071 704.437950 0 ] (cb9-281) [ 390 0 R /XYZ 99.933071 685.490928 0 ] (cb9-282) [ 390 0 R /XYZ 99.933071 666.543907 0 ] (cb9-283) [ 390 0 R /XYZ 99.933071 647.596885 0 ] (cb9-284) [ 390 0 R /XYZ 99.933071 628.649864 0 ] (cb9-285) [ 390 0 R /XYZ 99.933071 609.702842 0 ] (cb9-286) [ 390 0 R /XYZ 99.933071 590.755821 0 ] (cb9-287) [ 390 0 R /XYZ 99.933071 571.808799 0 ] (cb9-288) [ 390 0 R /XYZ 99.933071 552.861778 0 ] (cb9-29) [ 129 0 R /XYZ 99.933071 780.226036 0 ] (cb9-3) [ 99 0 R /XYZ 99.933071 574.796389 0 ] (cb9-30) [ 129 0 R /XYZ 99.933071 761.279014 0 ] (cb9-31) [ 129 0 R /XYZ 99.933071 742.331993 0 ] (cb9-32) [ 129 0 R /XYZ 99.933071 723.384971 0 ] (cb9-33) [ 129 0 R /XYZ 99.933071 704.437950 0 ] (cb9-34) [ 129 0 R /XYZ 99.933071 685.490928 0 ] (cb9-35) [ 129 0 R /XYZ 99.933071 666.543907 0 ] (cb9-36) [ 129 0 R /XYZ 99.933071 647.596885 0 ] (cb9-37) [ 129 0 R /XYZ 99.933071 628.649864 0 ] (cb9-38) [ 129 0 R /XYZ 99.933071 609.702842 0 ] (cb9-39) [ 129 0 R /XYZ 99.933071 590.755821 0 ] (cb9-4) [ 99 0 R /XYZ 99.933071 555.849367 0 ] (cb9-40) [ 129 0 R /XYZ 99.933071 571.808799 0 ] (cb9-41) [ 129 0 R /XYZ 99.933071 552.861778 0 ] (cb9-42) [ 129 0 R /XYZ 99.933071 533.914756 0 ] (cb9-43) [ 129 0 R /XYZ 99.933071 514.967735 0 ] (cb9-44) [ 129 0 R /XYZ 99.933071 496.020714 0 ] (cb9-45) [ 129 0 R /XYZ 99.933071 477.073692 0 ] (cb9-46) [ 129 0 R /XYZ 99.933071 458.126671 0 ] (cb9-47) [ 129 0 R /XYZ 99.933071 439.179649 0 ] (cb9-48) [ 129 0 R /XYZ 99.933071 420.232628 0 ] (cb9-49) [ 129 0 R /XYZ 99.933071 401.285606 0 ] (cb9-5) [ 99 0 R /XYZ 99.933071 536.902346 0 ] (cb9-50) [ 129 0 R /XYZ 99.933071 382.338585 0 ] (cb9-51) [ 129 0 R /XYZ 99.933071 363.391563 0 ] (cb9-52) [ 129 0 R /XYZ 99.933071 344.444542 0 ] (cb9-53) [ 129 0 R /XYZ 99.933071 325.497520 0 ] (cb9-54) [ 129 0 R /XYZ 99.933071 306.550499 0 ] (cb9-55) [ 129 0 R /XYZ 99.933071 287.603477 0 ] (cb9-56) [ 129 0 R /XYZ 99.933071 173.921348 0 ] (cb9-57) [ 129 0 R /XYZ 99.933071 154.974327 0 ] (cb9-58) [ 129 0 R /XYZ 99.933071 136.027305 0 ] (cb9-59) [ 129 0 R /XYZ 99.933071 117.080284 0 ] (cb9-6) [ 99 0 R /XYZ 99.933071 499.008303 0 ] (cb9-60) [ 129 0 R /XYZ 99.933071 98.133262 0 ] (cb9-61) [ 164 0 R /XYZ 99.933071 799.173057 0 ] (cb9-62) [ 164 0 R /XYZ 99.933071 780.226036 0 ] (cb9-63) [ 164 0 R /XYZ 99.933071 761.279014 0 ] (cb9-64) [ 164 0 R /XYZ 99.933071 742.331993 0 ] (cb9-65) [ 164 0 R /XYZ 99.933071 723.384971 0 ] (cb9-66) [ 164 0 R /XYZ 99.933071 704.437950 0 ] (cb9-67) [ 164 0 R /XYZ 99.933071 685.490928 0 ] (cb9-68) [ 164 0 R /XYZ 99.933071 571.808799 0 ] (cb9-69) [ 164 0 R /XYZ 99.933071 552.861778 0 ] (cb9-7) [ 99 0 R /XYZ 99.933071 480.061281 0 ] (cb9-70) [ 164 0 R /XYZ 99.933071 533.914756 0 ] (cb9-71) [ 164 0 R /XYZ 99.933071 514.967735 0 ] (cb9-72) [ 164 0 R /XYZ 99.933071 496.020714 0 ] (cb9-73) [ 164 0 R /XYZ 99.933071 477.073692 0 ] (cb9-74) [ 164 0 R /XYZ 99.933071 458.126671 0 ] (cb9-75) [ 164 0 R /XYZ 99.933071 439.179649 0 ] (cb9-76) [ 164 0 R /XYZ 99.933071 420.232628 0 ] (cb9-77) [ 164 0 R /XYZ 99.933071 401.285606 0 ] (cb9-78) [ 164 0 R /XYZ 99.933071 382.338585 0 ] (cb9-79) [ 164 0 R /XYZ 99.933071 363.391563 0 ] (cb9-8) [ 99 0 R /XYZ 99.933071 461.114260 0 ] (cb9-80) [ 164 0 R /XYZ 99.933071 249.709434 0 ] (cb9-81) [ 164 0 R /XYZ 99.933071 230.762413 0 ] (cb9-82) [ 164 0 R /XYZ 99.933071 211.815391 0 ] (cb9-83) [ 164 0 R /XYZ 99.933071 192.868370 0 ] (cb9-84) [ 164 0 R /XYZ 99.933071 173.921348 0 ] (cb9-85) [ 164 0 R /XYZ 99.933071 154.974327 0 ] (cb9-86) [ 164 0 R /XYZ 99.933071 136.027305 0 ] (cb9-87) [ 164 0 R /XYZ 99.933071 117.080284 0 ] (cb9-88) [ 164 0 R /XYZ 99.933071 98.133262 0 ] (cb9-89) [ 194 0 R /XYZ 99.933071 799.173057 0 ] (cb9-9) [ 99 0 R /XYZ 99.933071 442.167238 0 ] (cb9-90) [ 194 0 R /XYZ 99.933071 780.226036 0 ] (cb9-91) [ 194 0 R /XYZ 99.933071 761.279014 0 ] (cb9-92) [ 194 0 R /XYZ 99.933071 742.331993 0 ] (cb9-93) [ 194 0 R /XYZ 99.933071 723.384971 0 ] (cb9-94) [ 194 0 R /XYZ 99.933071 704.437950 0 ] (cb9-95) [ 194 0 R /XYZ 99.933071 685.490928 0 ] (cb9-96) [ 194 0 R /XYZ 99.933071 666.543907 0 ] (cb9-97) [ 194 0 R /XYZ 99.933071 647.596885 0 ] (cb9-98) [ 194 0 R /XYZ 99.933071 628.649864 0 ] (cb9-99) [ 194 0 R /XYZ 99.933071 609.702842 0 ] (configure-default-service-account) [ 53 0 R /XYZ 84.259843 561.590928 0 ] (configure-etcd-user-and-group) [ 45 0 R /XYZ 84.259843 283.686971 0 ] (configure-kernel-runtime-parameters) [ 45 0 R /XYZ 84.259843 539.222079 0 ] (configure-network-policy) [ 67 0 R /XYZ 84.259843 599.840928 0 ] (contentsbox) [ 8 0 R /XYZ 77.509843 756.850394 0 ] (create-etcd-user-and-group) [ 45 0 R /XYZ 84.259843 169.788971 0 ] (ensure-that-all-namespaces-have-network-policies-defined) [ 67 0 R /XYZ 84.259843 564.092928 0 ] (header_bottom_text) [ 6 0 R /XYZ 85.009843 422.718898 0 ] (known-issues) [ 45 0 R /XYZ 84.259843 799.370079 0 ] (overview) [ 30 0 R /XYZ 84.259843 272.631496 0 ] (reference-hardened-cloud-config-configuration) [ 534 0 R /XYZ 84.259843 537.158799 0 ] (reference-hardened-cloud-config-for-red-hat-enterprise-linux-8-rhel-8-and-ubuntu-20.04-lts) [ 557 0 R /XYZ 84.259843 511.863778 0 ] (reference-hardened-cloud-config-for-suse-linux-enterprise-server-15-sles-15-and-opensuse-leap-15) [ 534 0 R /XYZ 84.259843 397.010799 0 ] (reference-hardened-rke-cluster.yml-configuration) [ 77 0 R /XYZ 84.259843 191.441066 0 ] (reference-hardened-rke-template-configuration) [ 390 0 R /XYZ 84.259843 518.211778 0 ] (set-automountserviceaccounttoken-to-false-for-default-service-accounts) [ 53 0 R /XYZ 84.259843 522.092928 0 ] (title-block-header) [ 6 0 R /XYZ 84.259843 502.818898 0 ] ] +>> +>> +>> +endobj +4 0 obj +<< +/ExtGState << +/a1.0 << +/ca 1 +>> +/A1.0 << +/CA 1 +>> +>> +/XObject << +>> +/Pattern << +>> +/Shading << +>> +/Font 632 0 R +>> +endobj +5 0 obj +<< +/Filter /FlateDecode +/Length 2578 +>> +stream +xˎd 602%R^8 ;0 dad1.'8~~*]럋IDHGRU,x{Η7' 5 ~wCw|b5؊.j˩ ^4ϧ|v5Bk%|Ǯ2 kLT *JT9ZS.DUb#T0#WK0Úg Q`Fј +fĖ+QE1P 3MDs(t)-5ŚN5j1Gk &2lLze[PlI^*lIFt[]NuEDkrs3kЉW%!0]!7갣H]-Er:ؒU)l1ZnM}whd [RFu +lZBc@5XK \9@VeH-+2DRd P- +#FbعT[c&؂%2(hknMZd,P$pa0(^@\Rc,P0WNt+cKb,P`W cjQ UŕT[cX`Fb%jJHX`nr1wSEݔ,r)Kb,0p7Y*p] $t%e,BBZXȞBV 2Gu-Ċ'b) TX]  2ݘ0dpyCf,;-DR]  2 ,%D P] pXAXP]XP]ɪCCuE=1 ۖi$AHtD0Ӂ"1pMƂR 9c ȂcAq XP@J@;3T YT[cд9cAMESb, e`,@FZDMwd YfXPsA=YeWX jP PKǦg,CO `t j-*E2sI6c2]bd,[ 2xrTE QKbΤיXVBO MPiT[ O/3c(sh6Mg,p-Ic|כ lAXФ&3Љ071`_z +Mxd6-ެ`KFBuIS#LB :1]-cA+כX4&2zz$ެ db:pia,@&2[V +3^q:=17##{*=3HfBO VB E>fL>GTP3l~|췟OF=O'ej֥\ov~~\?$s;Qeц2=cb7 }᧞<<||tU_- }F8OTOoN}_bO?BEhc9m]n_}?U>uy~}e?" $ȴ|70f\'wqQ_7u~:q'L+W*ؑ{,'.ږKF+w}'ŗ*_O q0 ȋ/W}[Dzo;UIMX[:hvYRcz1!+|\zӥָ{, 6>"lƞ\ϲ#`bl`@\HqʼJBv9ExZ{޶ys-IYqY1Hhu_LGqzAx`zki +6놶]7\~PU6a~6'^?̩+K8Mߧ.n?-՗g*PVcvL7ڔs.u"ˉ_ɸa1~[_>f6DJץm2ex";&mW`57Ax9!)W;/VH u,YR䟷 WT)9ޠ(jf }oN$2clwX} ߡ;{nSkuHMx܋)3Y e8OW~/;ܪY`%n}Lv>:W҈ԥf[[7[mw?y{u\҉)}ޥtKh9zy {t&9kknR_:}PAGLlޱ1-;:s%e鼯ʜļJy;"̖t&@yw纞NVϙK:)Sql{D )+3uY~IClX:N00D7XoGq> +endobj +7 0 obj +<< +/Filter /FlateDecode +/Length 3519 +>> +stream +xMo$+l`e #6@ +!?Ş&iRZ$]k=d%(N?5[Oj^џ8n&zbN-'j]p>~/uźoN'iz|2U'Y|XC6Y˧hZC//m!5+Yk:x HA 2tkX'p$ :(ͅZ}Gm'CX-=H/?Vb<8JĵNj٣P6 ]ȑ[-eֱd'U3yզzDy2]XQ"zyPzH}&D-'< :fb":uED'r3Dt"FQ|FwDt":}n}DY(f_EQ? $%#\tT!ʷYiɵ5TQ DAP=6 ]F#Տ,iLCy.J&ረ̃`QKtQСPE F="ʑhQ ND'DobCgDщD݉Dt"j](E(׈Za7 '.rs KDW$ nG +?4X0&Gۦ`y#Տ7&[!;Bw":F7>Kػ(DQ:7@S-+>TQ DAP6ΣGD9Rx˒d< + dꎈJ @tˤM$p, Hp =>/?D4΍!(TM7x0=rd;Bw":F7>KꯢMII)u+M$p, vn4] +:jѶ3~eIcECuQ2HU p|;C!<% *y0bnT?/@F0@w"db":ZT *z.F:#ND'ohotGND'W:_EY(_EQF48NIGO10#\.vn"chGC[ 1qt%)vn GZm:<9Rx~cECuQ2HU #!n I"4{ȃActC~ <cE!щDע}h":};ND_t]֟>takX7K:$]OGo?|ӏ[PJ?s;_44_dԙ;2W/9-mH5dzE+(Y_wm:%۰KՐnB+s9ǐu]R?`טG/Bh +Pn?^Xϡj.'SDĢ՟Ab za΍<¼VXM9K.;sc$vBpR%Eu2T8@gdLpkXzBj;Uf 'I,aeǸ +=.?_Tt~IGDOmH,y1{#]~EgWS,0'/-ϻwܯ*+ {a2Ly=<[̷-/^s[ou*mK} +}tR~򉊕6UqYYt'x*$S㫷epa3\[IjڶcqoNTkL.H&gZG.WE_f6CWMmjiތ6_GS4|iߔb8tfG>ȔVjId۲A>IY[? +ܱLִu ׶&B:'rkjtLQMЩT_[TEwv>dͬiS-$ڮ#Ov52[ħy +M-mѮ4q<JjU1|*A?iyP69)Vc"4V˿7< ZIńG줰d$Cu\/En 1x:ˤZ Bz̵{eju&)Ƨ͕( rDzo|/doRWgACf{c˸ ׁ5_󶰹k~NX6v DM +,4zɕ6;KOeXor^7J_j=}:nT-L[+1I٘Ć ϲNjh10_;>]^[6.cikGUF楹rhȍK+ráY^0IG)/.]!83R褅a*l(s̢m&NN N,x8{C%mjo=m`Em^_amͱK9r- ٶa4xk(Oxt&օM5)հl#N(Ŷ`7Ǭ zJ8-vvKm> +endobj +9 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 76.423228 671.061024 504.679134 638.361024 ] +/BS << +/W 0 +>> +/Dest (overview) +>> +endobj +10 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 489.361263 663.111024 496.429134 648.111024 ] +/BS << +/W 0 +>> +/Dest (overview) +>> +endobj +11 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 76.423228 626.361024 504.679134 593.661024 ] +/BS << +/W 0 +>> +/Dest (configure-kernel-runtime-parameters) +>> +endobj +12 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 488.881526 618.411024 496.429134 603.411024 ] +/BS << +/W 0 +>> +/Dest (configure-kernel-runtime-parameters) +>> +endobj +13 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 76.423228 581.661024 504.679134 548.961024 ] +/BS << +/W 0 +>> +/Dest (configure-etcd-user-and-group) +>> +endobj +14 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 146.805601 573.711024 177.256773 554.961024 ] +/BS << +/W 0 +>> +/Dest (configure-etcd-user-and-group) +>> +endobj +15 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 488.881526 573.711024 496.429134 558.711024 ] +/BS << +/W 0 +>> +/Dest (configure-etcd-user-and-group) +>> +endobj +16 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 76.423228 536.961024 504.679134 504.261024 ] +/BS << +/W 0 +>> +/Dest (configure-default-service-account) +>> +endobj +17 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 146.805601 529.011024 197.845152 510.261024 ] +/BS << +/W 0 +>> +/Dest (configure-default-service-account) +>> +endobj +18 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 488.893245 529.011024 496.429134 514.011024 ] +/BS << +/W 0 +>> +/Dest (configure-default-service-account) +>> +endobj +19 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 76.423228 492.261024 504.679134 459.561024 ] +/BS << +/W 0 +>> +/Dest (configure-network-policy) +>> +endobj +20 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 488.809749 484.311024 496.429134 469.311024 ] +/BS << +/W 0 +>> +/Dest (configure-network-policy) +>> +endobj +21 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 76.423228 447.561024 504.679134 414.861024 ] +/BS << +/W 0 +>> +/Dest (reference-hardened-rke-cluster.yml-configuration) +>> +endobj +22 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 234.738707 439.611024 313.229430 420.861024 ] +/BS << +/W 0 +>> +/Dest (reference-hardened-rke-cluster.yml-configuration) +>> +endobj +23 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 489.877620 439.611024 496.429134 424.611024 ] +/BS << +/W 0 +>> +/Dest (reference-hardened-rke-cluster.yml-configuration) +>> +endobj +24 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 76.423228 402.861024 504.679134 370.161024 ] +/BS << +/W 0 +>> +/Dest (reference-hardened-rke-template-configuration) +>> +endobj +25 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 484.970394 394.911024 496.429134 379.911024 ] +/BS << +/W 0 +>> +/Dest (reference-hardened-rke-template-configuration) +>> +endobj +26 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 76.423228 358.161024 504.679134 325.461024 ] +/BS << +/W 0 +>> +/Dest (reference-hardened-cloud-config-configuration) +>> +endobj +27 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 210.896178 350.211024 288.867613 335.211024 ] +/BS << +/W 0 +>> +/Dest (reference-hardened-cloud-config-configuration) +>> +endobj +28 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 481.993831 350.211024 496.429134 335.211024 ] +/BS << +/W 0 +>> +/Dest (reference-hardened-cloud-config-configuration) +>> +endobj +29 0 obj +<< +/Filter /FlateDecode +/Length 4947 +>> +stream +x]K$qׯ#6l/5fJ#EV fFfUuw;`DazQdw<ְ8y$ 2r 0jX\\FlH%@ 55, 0A[.awaeB@Xs^ p,JM&K԰ wmFT^lǕ'cѰ wPy p5,OXQk=Vjܵ p}Cа0Y4`}Aꪼ2 Xr`L4!ɠհ w "p5,]Ch{TtiX 3FÂb34,@]~3kXP~[?^~3?)k], _Nh&]o%z[Y:&Izd`'uG3]H~ztBӹaیKٍ.칎fQxJBuԹsti>S57q|SXqLDh_@N|sBs5y:w[ v'8Z.,3r\;\@[lK6@r䂠7ŌS/_Tc)3\U/'^#ׁ^LK9>/0(ɻȟubwXvmò4݇buKY0nnfŨGZxhjv@vH9taK Sæ^c>={ -=s挶lNGM$G$`~>je9зª=W#0j|;`Cx6__ }/]+5a{tax(V]oנ_ =|z7|5ѷ|{=bwх&|5ix%-*/?/?A :dLO nB8fyadͿۘyO9<1e7,ןqA`&sG_?(qI^lzY2W|C/rK|gT[MFBv%V}T5՘}ZtXb^|y%?Ycw|7;-?5N2wRץj-վ5E_z>VZ=-Ɯ7 ~/84߬꫍kϏ;Tgdk՘ +zmӡ?tŦ'l6|ӕuz_g\Af4S<Ύhh(淈ITiAMc9m16&9$3ṡ#z٩%U-<6fX j]>+m⸉ϡ/5^2[ֈU<"[O +xW& < 5js;sY;kώ6qJ9Zw|޺ռ/&{ >SG֖Cŵzǭ.?ظe_֡FMP[$߷n9Ĝ{ĪZCuQfsxc3>?.;_?8}n}E%b$_f+%pbhͷJ-{t>aպouO6K.Go?mx%s_/G/}- +r/t0Dn =ȣFO_Pο_h{9X`8Iuh )n)21"+nlTI.P9_P~7O8~LlhPE]dO`_ a&"a@T^r>sTbL%3fغo<8DydJ9̺g*8U:NyWXh<ݫnVMխZYmjsVMMࣤ?_MlgA-i-" y +Jﲆ{7=b=|hQ<\I˞0 ZL$oA9DM/Κd3aDU%>S!' +Ɂ-BJȣo^9^x[j<6 Zx}kMݕn|F"A1D=&r61sCb&~ ObLlJ&_ȯ녽_^Q_T O;9.|$zh;M6 bmuڀt +¶2x6)kqCz|B=8vWssom>}693:_[/ l9|e^<͒So|fG/*kVx:# -%}J5gtL-UfwMk uC_ŵNylAayC2֢7x7F~aONk砋֙5}lq+tKҥ/,GyA~"B3*%_s;4<~X +Hҟ1~zZMɍ?aio8{MLo817&s '>+X 'rGګF!wk7f4v4;ƛ}Kߣ=V7$p|(pk6}徨҉ R9󻢲d{k9J#sK/Nf[~}9BS%MLή^to%LO[j,.'L+tOx|0TkPo7 ivn+D֛.d~+pr*!VAIwA [u5NlKY:%#izU.*zkeyTiPGPBw&En7bDm\夎[F~kT9^])hל1:#GƚHn'^NZ%v2:t0$z!ҩ)WW4+?ߠ"4*# / +W}IG'IzlC,>V˦rVfgmpRcj1m˺FfiX)*iR:j0mhE&cΫ$qO75}\x6Z9si]LbN0zɵZ:K믽ȴri^?ڎvewd3':~];5lr%㷳Jle#w}lq^St`#mCP۸ٵu τeȼ56}]jwh5B\}I> }IR(.ńk6 )'igI\B|;2+EWl ) (JEX+ *06ө~e[s/jv5Y^ԞoMN/Zp!sTPsmTiäuhp&1Ur[ulqVZOѩ2*͉AF%nƲܺnɷ(tN:G^2+(nT#}mqڞ35*{MGڨJ_zG!/*OőqlO2٣&Yt` ![FmŨ&=j+r.oߧf~~2;qsZP +endstream +endobj +30 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 29 0 R +/Resources 4 0 R +/Annots [ 31 0 R 32 0 R 33 0 R 34 0 R 35 0 R 36 0 R 37 0 R 38 0 R 39 0 R 40 0 R 41 0 R 42 0 R 43 0 R ] +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +31 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 84.259843 445.281496 409.271317 430.281496 ] +/BS << +/W 0 +>> +/A << +/Type /Action +/S /URI +/URI (https://releases.rancher.com/documents/security/2.6/Rancher_v2-6_CIS_v1-6_Hardening_Guide.pdf) +>> +>> +endobj +32 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 107.173228 418.281496 162.214732 403.281496 ] +/BS << +/W 0 +>> +/Dest (overview) +>> +endobj +33 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 107.173228 403.281496 332.953990 388.281496 ] +/BS << +/W 0 +>> +/Dest (configure-kernel-runtime-parameters) +>> +endobj +34 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 107.173228 386.659914 295.486949 371.659914 ] +/BS << +/W 0 +>> +/Dest (configure-etcd-user-and-group) +>> +endobj +35 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 169.605601 388.281496 200.056773 369.531496 ] +/BS << +/W 0 +>> +/Dest (configure-etcd-user-and-group) +>> +endobj +36 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 107.173228 367.909914 319.724254 352.909914 ] +/BS << +/W 0 +>> +/Dest (configure-default-service-account) +>> +endobj +37 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 169.605601 369.531496 220.645152 350.781496 ] +/BS << +/W 0 +>> +/Dest (configure-default-service-account) +>> +endobj +38 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 107.173228 350.781496 255.796275 335.781496 ] +/BS << +/W 0 +>> +/Dest (configure-network-policy) +>> +endobj +39 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 107.173228 317.031496 400.593883 302.031496 ] +/BS << +/W 0 +>> +/Dest (reference-hardened-rke-template-configuration) +>> +endobj +40 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 107.173228 302.031496 397.449596 287.031496 ] +/BS << +/W 0 +>> +/Dest (reference-hardened-cloud-config-configuration) +>> +endobj +41 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 233.696178 302.031496 311.667613 287.031496 ] +/BS << +/W 0 +>> +/Dest (reference-hardened-cloud-config-configuration) +>> +endobj +42 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 293.575760 126.231496 442.798661 111.231496 ] +/BS << +/W 0 +>> +/A << +/Type /Action +/S /URI +/URI (file:///doc_tools/%7B%7B%3Cbaseurl%3E%7D%7D/rancher/v2.6/en/security/hardening-guides/1.6-benchmark-2.6/) +>> +>> +endobj +43 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 84.259843 111.231496 285.368241 96.231496 ] +/BS << +/W 0 +>> +/A << +/Type /Action +/S /URI +/URI (file:///doc_tools/%7B%7B%3Cbaseurl%3E%7D%7D/rancher/v2.6/en/security/hardening-guides/1.6-benchmark-2.6/) +>> +>> +endobj +44 0 obj +<< +/Filter /FlateDecode +/Length 5107 +>> +stream +x\Y%7~?<#㥼I DEbar Bw>_u{݉h2m\Ujiwşw+L)5':_>u&G;6K7dzMt5xĄMt~9Ky?9sǝ5y(\rQih.(M1}LJSrKUk1cSTKԚy_6eMlHkƬ49 >Vm#(xE ygRmm.{-2贶jRJhm^bNk%:" +%km ^|AkC:Ui+U6buZ[ &BUْ˱hb jmΙ jvKm%뵶^6D ezE}& @^lN`% |Ά' | + |r  5֡; /UKaA7hXx4,@[hXdӰ oɺaAMaA→HÂȅ)U 07;5,]laAꆘ$ p7p7T X  pkXaҰb|)VbI e=4, ஏX湁>Zaw=a  w!Q V{a3a*ܟVƕ4,]W>!qeBƥT5,]8aA>x5,]ੵ6jX0hXfmZ X [$j[60$aXġ$.{հ wm AÂܵGm/x נ VV@ + k4wUÂeaALE0h{ R!SsNjцOmKQÂlA, P& XjѰ {װ {AD:Djz^`.j~@@2S*9&c֖)paAdȥLװo8\Ұl͑l L̵6^dkXHmiXPOJY`'!SWffK ]Xl:tVvc6pyp[3s,kH~ṋQ|7/H9߯__bO)w'7"5/_ Jo_lkUkEqMAqD<6:&a4cy a_c1QIqU1Y)`[ZD[I)xʒ>B4`%5~jElk mwyA%NR8:x+SaY2[/^-ݴj뙽)BF!7f YGY\Abqwb˴ ֚Dw|m)t_;W7)woo6>ܴ-j&bme!s&ϋ [MǹVznʯp$αv= 9trkPN;pWY|u_CmE_]`Kq |Wy1zيk-]쳮b!fw+/ۯ~Lw9̣fs=4]VѶǦ--tu`W=Pn`n9HÃj+?Ȋ4kxC< 1H [SGpODT@< 1%SLG{˵QԂbE8݀JpLunب Ǡ埄֘ibHE>U}^&ncj7r|襝޾?OoR~>+ľ_n5}/& IdL)kۦc^tKs#?Rf2;`Z +?4YssOy6@/sm޹E[jm}v}`kcW9N>K|r+J#sY*p,c/gL1n}k_{[p8a3LY'dלD]K!C|} #_%SXeP>oEwq^/YUǡqkeIP@3`>}޸$n}EbxF>Puɸt!~l PWpݯaDvHi|qcX/~۴u{;bOE:J ec}&IWk2*7^,J6 䎲;R :tC;(T`U TkEICl"MI|Rl~]{1=RXh;iDzg +|~'UnU1 q(:]l92#W#nZGcg"ĔtgkRB09U-nJ[v+5:5Qub7Gjn(*dzn8+7.|E=gTx7mxtvEis_ +d|_==߅SrPd[!F^ hyDM7w*GK:_l +^•->c-ziStݮ,mC=AYKc]+,m#!jSS< ;Q:qA:ʕ\`+[A7cзG)s݉?E7N\9󔳮&lk;Ҫ̊>nƝ{>o +bhYw-ׄ, ]U߅/> ۔9DJHjt39nGnIIVXU¬h[ǭOIoͳs>fᴌ}.o_lntq|VZ+VB88z//G t2B'OhCZo[FQFJ@7vek/D\R:ͽ3X}v߷;\(a=ѱrg1^<ːbȾ>2T3xmN\pm{ݣQՏYw;]󟴼tX/0˯Eb&n!Wz.2Ȯc7ŃEݿ'DdRpQم\|F +ӎ m?g}%s>ލ_G|:BDd^5nńv+1uAw8;%IGMΏO ѥIK9~B {JItvZLAΡzRHB|gev5cc*ݐhW} +rRkš<*k풟LSe^]D-7*ܒkxi7@"8/ +Ibr^&Lf +TAeY~t}y5'y+$IOuHqp*yج":~q؆*D&: ؒݰAfX)3&9S~ِ'qlhd scIN;_$9;w+l/c=MI&d8(Lїw@'bf97q/7Crci2:r➯':ݿαqoZd|Ǫ~_>[wSꋤ1uN e^E}N {a9skЂ6Fn('NLŀ ls)n],ˊ*> +endobj +46 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 191.048717 658.622079 244.661998 643.622079 ] +/BS << +/W 0 +>> +/A << +/Type /Action +/S /URI +/URI (file:///doc_tools/%7B%7B%3Cbaseurl%3E%7D%7D/rancher/v2.6/en/admin-settings/pod-security-policies/) +>> +>> +endobj +47 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 439.875057 101.433071 421.125057 ] +/BS << +/W 0 +>> +/Dest (cb1-1) +>> +endobj +48 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 420.928036 99.933071 402.178036 ] +/BS << +/W 0 +>> +/Dest (cb1-2) +>> +endobj +49 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 401.981014 99.933071 383.231014 ] +/BS << +/W 0 +>> +/Dest (cb1-3) +>> +endobj +50 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 383.033993 99.933071 364.283993 ] +/BS << +/W 0 +>> +/Dest (cb1-4) +>> +endobj +51 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 364.086971 99.933071 345.336971 ] +/BS << +/W 0 +>> +/Dest (cb1-5) +>> +endobj +52 0 obj +<< +/Filter /FlateDecode +/Length 4930 +>> +stream +x]K$9nDs0l/<^]`COkFYQzD33݀P HJhwOfgr.)˯kRWKl.*M,'\9\%J)^.]m븫7DodRNUV&%ŠKX#$|d%%12 N1|r +!),g!֪<b+<Lң\|Qy <#21 Nc Y1d : KpVA4^,ɪXaw=a  w!A V{ ɂY*pb XB%kXKT3s,(//+ecM? xej{\Wc∄s !`q]t!n˵q[!;MHƊ*uI֡#/yjYGS{һkvxhIJ'IbRq 4՞Jd}λ5݀ p^ SR{һkvx2?M+G^GG?_YIfd|"rDhu#_Aʄo\wYyeW6k[9-ˣ_?0߃,;E3AĖKjt RPͶ;zAvc9U.~ʯ0E"L㍀avɦ.C Alq9vd +_$l"d2X䘸eCKL@XiZvClJQ&ŇcR8h d1w7u-o1,yehz;ߜ. k,I#,QM#٨:7J뢒~(tye]eU_?TIu\&N=2Tle2%v~N3^r~q LP'4f/yA?Wk7"p7:}NAwyZӻۍ.]5a +49CdtRT''nSsN2t:ۗLSSTZr!d_e:WۙZ9 =ž p+h_i p2t~qg1 CuڔW/@9MAZjX w?7!;1]@-Y5K-uC䭉Tc۬26>Y`mۼ}Y g tYxMn6r)M?bzcAx{e异"C^GU y-9Z/$E@G*_z8ea7"'_-{:$v\ sbQ(nSU q4ѫW(]?oaˀ"[Gr =/M܃;Y:9:mwZwɡ Lm';~""?ӆΌmkÝ#pw#ݤHvl%9:W1)!''eCt2'aV2C'ģeX/bśkѩSszxD#e#xRO??O:i뫹Lu'N.)UyL6M) +*gWom?rzSgr1r]hC]) +݇ +6Pg|&ŗY +ntF3j柲7?=7[ޯNzmMf߲b^n|"56|WguSrGڱ~פǻ/[{v6}̰ 66l?R1|<م_\ǹ?o^璼"ߣm^58Jsf>.459vHLa'*z&8 ]6_0}3c[uMNF=ĘQo) &X ~өʍu$OzƎl6Dz{^" ,dyl7/=' iGӷdo*$9 YU©=ɐ:(@&'?2[l['|3h+=:@*;leB(%HsQjcTn}R.*ȅuțM icrbiam;85pN^t6n9gr<3&|3Pv?]yssTus; %G{NvD^AA'gyOhyϱ[s:p +h)]A/􇊻>,$[{|vOm#b4)3yDE[ cPʯ~uV;o+4di іޭR}jv{uz NJRوUrBdKuU!zf}᯽+UZ:0\eovwױBfa~-v=Sk< T% +8|C=Q?jNէꗎ"jFw$[Vz(!6A \v48*>$K0HSw}1]uq4,?E|[STu5մm$TJ>N +i`v4j~siĽ\iE-JTUKe:x2JBiXj_dy,C0 \8oa` f"Mvr^˪*LF7_*B +/II,5F'27;y;갦i%K`tlt؆:l%nu,ӱr&&MmCr>UE7{r6SqyqF|d1:d& 0F*֎Nr}L/^YoP煯?W,~ێc<{2yk>IcOyAvy UIu;tA]0t8}7]hLC|i2_:6a/nN%@M*\̷goJx2%TgosW&ӒuqW20MNg]70cT_ jŪ%sWSߺ/&z7 tsf/(f +endstream +endobj +53 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 52 0 R +/Resources 4 0 R +/Annots [ 54 0 R 55 0 R 56 0 R 57 0 R 58 0 R 59 0 R 60 0 R 61 0 R 62 0 R 63 0 R 64 0 R 65 0 R ] +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +54 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 751.923057 101.433071 733.173057 ] +/BS << +/W 0 +>> +/Dest (cb2-1) +>> +endobj +55 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 732.976036 99.933071 714.226036 ] +/BS << +/W 0 +>> +/Dest (cb2-2) +>> +endobj +56 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 653.081993 101.433071 634.331993 ] +/BS << +/W 0 +>> +/Dest (cb3-1) +>> +endobj +57 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 634.134971 99.933071 615.384971 ] +/BS << +/W 0 +>> +/Dest (cb3-2) +>> +endobj +58 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 615.187950 99.933071 596.437950 ] +/BS << +/W 0 +>> +/Dest (cb3-3) +>> +endobj +59 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 596.240928 99.933071 577.490928 ] +/BS << +/W 0 +>> +/Dest (cb3-4) +>> +endobj +60 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 280.647907 101.433071 261.897907 ] +/BS << +/W 0 +>> +/Dest (cb4-1) +>> +endobj +61 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 215.950885 101.433071 197.200885 ] +/BS << +/W 0 +>> +/Dest (cb5-1) +>> +endobj +62 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 197.003864 99.933071 178.253864 ] +/BS << +/W 0 +>> +/Dest (cb5-2) +>> +endobj +63 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 178.056842 99.933071 159.306842 ] +/BS << +/W 0 +>> +/Dest (cb5-3) +>> +endobj +64 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 159.109821 99.933071 140.359821 ] +/BS << +/W 0 +>> +/Dest (cb5-4) +>> +endobj +65 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 140.162799 99.933071 121.412799 ] +/BS << +/W 0 +>> +/Dest (cb5-5) +>> +endobj +66 0 obj +<< +/Filter /FlateDecode +/Length 4895 +>> +stream +x\[$~_Qφ1 ֻͮt9cB"34]RE(q3ԜzŚ-kgl?ǫMt5x !\v_CyuvzSk.^YL.9ۨ*XǤ3Xb bL 1 y_V GV>;X՘ykUD+<MғΤR}Uy< L.:WMJ)G%V4^,YY(d )XY|Ae⡝ +dwYlN`b-D*gK.GxΙ *#ʃ,0>A øaP/P/PA_T:Ӏ X)iHd!Dʃ,CaASа m ޢaAܒNÂ%뒆 1J"  T5,԰ tY 1y `7PIn g5, n X  pa5,U 0iX_1aw}aw}F +Py K27pG5, '4,U. $djX|`2QhX")1 2ԤaAj\ƕKjXHTAÂ| kX. SkhXP(.5,@` Ty0aÛa6?lb KaA)EÂR ^&7BT|T V +4,@\~hXP~[<%nkHT,%ʃ,($7N/A_IÂ7(3DE +7. +dx]TI`V[ǫtfgQ;[Ֆ"[eg`GR;˒Ԯ*Ϊm E=FLXOE9d^dEX瘧;- syU?(8^6܂Q8T~hǭm7 ^2L[m.͂Bҥ}BڣY*$H.~߿zӯH1ZK~7v+~7U̕9mS;7~w`bSWn,t3&+To[cvC>U؉ +(-,UÀRfwoO]18B_ѭaѡsNCW.Y>>pM4'_ iyucĵMq4wk|AkВז/.A=Փ3=/ @ -bm`8q?OleYbHwc?'.K5n*%:'݈e=+!񓍎аҏ=Ǒ_u18t!yQy7#0 +kfZ7/\ϳ+_]1K|zWw{TȌaǮ<$O +Spl16`V.拫c\ԑԶ,Np|*ƺyqߝ9]‾ݽ֜[)ݓwO7oms2}~E_9.Wi-]y=~}(qmr,Q.<i9_~+CXg@>)\5 p>r|`wrwSZ]?sg)/DM:ltiڿIJ~ٹJ.Dⷤws7fAU@.&.phMM6 UxߵCwQ7S_RX׋4;8J +`_*G9{e=o{d-۰ۣ>WYĺݟϯ(MHѕ'/Xьf + 9ZwbԊRG@PMzO더0 0j ?mp||#ύH QMx +B6n +ಛ.eΕ?mOxe?̕P.$F7 +YN}!ת UP4.޴g궇TRܭq`ݮz=cW֮ #~r=/Lzd.>սCy(Gm2ںX=S!n7nBcykF\!Ҝ'V"lҧIIq|q m[ O:߮ +B;_zTgq?J? !jUΧd|}|u{'q)u,Xc{}I񛎥97Q s#[l꺻xw{t$[9x'#㯎S=6zo\>7OHtOuH䩇OV(걡]zL{=>kW)Lݮi +ӷ6~w7mN49OuX} +KlZx7k_}z*/{ ;M^a`iȇ1rf"VxKڬ\b*mCy֌_\P\|_4b|w֦9cPp&o]Oj^ЭrqFB'E̩&=_]^н?;N=YE>Tߙ󙺋N{zv{MLmz^Wvo7VMO +쾣tU~¦ǹ^{Id7pOU1xo/zmK>Cy ?鞢SW>8}gQo1kz 8?+e~'= ܘQplw;{z{&\cn<\kF}ΛOto;K]`oEJkkUddow=|)kwiNvV/ऽyLRлC{nE e ֗r#vŸX"5xʃP?t so\6o*?ӧvЂjgаB]7OM0 p :j?O!.ylqX*Z(CWh7`S*bĆsKNlnk[v4"q.1+glj(ļ\Iͥ +i8@ETFr}y4/±9[%IzCyPy} G3ʩ;B8919yѺGv_EN=:}P$u,tjhi{8rNAީ1WF3Cc9>*e[OiD14iceJ밮j)bR&5UJ:70 +(9y%BKr<3tڴv϶W Y9,}$vvV:o!Q >Aے0~hi{y=y%P6;.1CL?V]p&uˮom[Esࢷ'uz׳nZ'Ƥ +endstream +endobj +67 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 66 0 R +/Resources 4 0 R +/Annots [ 68 0 R 69 0 R 70 0 R 71 0 R 72 0 R 73 0 R 74 0 R 75 0 R ] +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +68 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 748.173057 101.433071 729.423057 ] +/BS << +/W 0 +>> +/Dest (cb6-1) +>> +endobj +69 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 729.226036 99.933071 710.476036 ] +/BS << +/W 0 +>> +/Dest (cb6-2) +>> +endobj +70 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 710.279014 99.933071 691.529014 ] +/BS << +/W 0 +>> +/Dest (cb6-3) +>> +endobj +71 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 672.384971 99.933071 653.634971 ] +/BS << +/W 0 +>> +/Dest (cb6-4) +>> +endobj +72 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 634.490928 99.933071 615.740928 ] +/BS << +/W 0 +>> +/Dest (cb6-5) +>> +endobj +73 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 287.395585 321.344928 323.513505 306.344928 ] +/BS << +/W 0 +>> +/A << +/Type /Action +/S /URI +/URI (https://github.com/projectcalico/canal) +>> +>> +endobj +74 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 84.259843 291.344928 111.294999 276.344928 ] +/BS << +/W 0 +>> +/A << +/Type /Action +/S /URI +/URI (https://www.suse.com/c/rancher_blog/comparing-kubernetes-cni-providers-flannel-calico-canal-and-weave/) +>> +>> +endobj +75 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 308.171464 172.723346 401.561844 157.723346 ] +/BS << +/W 0 +>> +/A << +/Type /Action +/S /URI +/URI (https://kubernetes.io/docs/concepts/services-networking/network-policies/) +>> +>> +endobj +76 0 obj +<< +/Filter /FlateDecode +/Length 5546 +>> +stream +x]K9rׯC2/`0x<^c@CFŢe`}L>23Ju &x| l{6RϗϧNfOybuSƿp!g!Pb. mnǟO_:즜R4JQFy?|P9kE`Gv/`sZ`Rq/`EZذGa1j12)ZQfʈz)Vgc2^e6ZxaJQ+  I+A:Y)K֕2bZYĬeӾ?Dd2k'$ %9jxdVFW+Nx))Ļɐ$f4-}^^;+B *tF#VlOW0uGr.$UΗ/ofZ!UNVx)V*Ԙ-eSy+ĵ!cnB% ^1QfKw$:um> " iOWԕn_} IuOG8l$:U:c H6 +ߢ?vy՟G&+ +:W"&7(N)!QFx*{+]Lc]L*Wz n%Y 켐^1ـ  2w?O~@wNC .]>wr3E$2q{#A)+r]V9?E +39Ime'dv;,vĸr׻?q!Q]h/f#hO8&Φdvv@SF-W\^)4Ƶ]BZZ?q߶˫ř}ݲd a᱈"-~W)}./Eq~?^ޯwV2đHM21jj ru~ + ̚Z7vSU`9d:-Ư`&]2QYmiwUy;Ѱujb/XGvozjH0ƻc%L/FߏP^#}׼0^vR*oGT&e *p^R$Wl}1B%2m-W9KO1WBk˪G{mq tGȞ4|W$[P/ǫFw+kN/BYx|W|_7N/{WSrѼ:׍>}tf9{䋁 +°԰V~,ke*,+Y~iX JsNtj6 ɾgh^|vY]Y/>^(-Rp߭j固E˂ckK*6EV5`b/_<)6y> In6e18u5c'vY>L\3Ole[h5Y7Sz㼆߬ya|`d}>TXڄHu$L&$}uB݋P9}}/^˰?D A0K0:=,PϫflDܗgwe<,nCjOע A4x|?pQ_!u/ LEA-:/ge#4b Y+{Cnw__].ۮpXN+n )dي)c‡I ˱1;Mu!-mv?S9!ݒ~y{ [pUZ kz9x/rIS/=,mgˢ֯#k _'#^Uب5]M_2_?73t\37iA!1|:Eeǣ!N9pZ; |D< &K#OY>qgc,X9֍xXxG%wW^_mw _6.Iq;c"8cTצݑ"Lmh'U8RDDzį~qOf)P)fGۓe}6CO)8aOПC0F\t)|[u^{6~\dc/^J iO/3T^kN'}&l%5S!KfTBjۋybf2}6|h'+mLE؍u)99_G9y]vrJ/=$u +1LmШEVA|o^ݎ„RAu1"wÛ?aLZW|wM{J`w›njWDYvh$[|˿8R#& +^PQ~M]c6jFPք +*[@85 mlh#1DYZG@ "FrE(Rw%A}Y}NWX79&,99[<_iFF>+۫y|5aϭ4{N=LYU^w +4vWp 4] gM|m,?\畗G~AY>];,_WP"8:;*z^=ۗk=\5ʚi;fF x\} gFNO1^MOZ'J TrUgHC`fnhKgHm+9ʩmֵP2?ueC{(a|-ϖnmauٹx7phIF(s&i1BzS?HF#Ǜ9|Cur~)dYߒOr~:+dd!.ŶQ]N)o$?'ԓ _FZMεcӦns +5ݓdA&ZBc`xHd]'gY.s9QtvP}A-Y׵*] ֕FSQ}})1i2[59?t&jHܞ9t9-؞JjtS.TJp} _a$@յIݸr*OW4t{T9dkV(PU\Xn&%w"k!P5 wvGE16eQmO7:pӚ aT,nsvO8ywkغ"#.t'Y~7H%6dZC儙IuӁ>iV:}@vtA2b dV7}o!9r%H62CyhwhQnQيM1ȱ&zGU5 aR/>̎MWZy4ތ Eo+Qbj/,Bmި#PT*a}*GUjm{Ԃ=h5j~+mi(\f*+OaVR8R-ԿV{Vڭ#^@~4'[,Zk7*'}:ڞH葩/¿/uBY{-361PiGw0bDNÔunqg211V~;3 +qTyzlWzLN?3BDN*oC# zh:cVS'zB +endstream +endobj +77 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 76 0 R +/Resources 4 0 R +/Annots [ 78 0 R 79 0 R 80 0 R 81 0 R 82 0 R 83 0 R 84 0 R 85 0 R 86 0 R 87 0 R 88 0 R 89 0 R 90 0 R 91 0 R 92 0 R 93 0 R 94 0 R 95 0 R 96 0 R 97 0 R ] +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +78 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 715.084475 101.433071 696.334475 ] +/BS << +/W 0 +>> +/Dest (cb7-1) +>> +endobj +79 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 696.137453 99.933071 677.387453 ] +/BS << +/W 0 +>> +/Dest (cb7-2) +>> +endobj +80 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 677.190432 99.933071 658.440432 ] +/BS << +/W 0 +>> +/Dest (cb7-3) +>> +endobj +81 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 658.243410 99.933071 639.493410 ] +/BS << +/W 0 +>> +/Dest (cb7-4) +>> +endobj +82 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 639.296389 99.933071 620.546389 ] +/BS << +/W 0 +>> +/Dest (cb7-5) +>> +endobj +83 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 620.349367 99.933071 601.599367 ] +/BS << +/W 0 +>> +/Dest (cb7-6) +>> +endobj +84 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 601.402346 99.933071 582.652346 ] +/BS << +/W 0 +>> +/Dest (cb7-7) +>> +endobj +85 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 582.455324 99.933071 563.705324 ] +/BS << +/W 0 +>> +/Dest (cb7-8) +>> +endobj +86 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 563.508303 99.933071 544.758303 ] +/BS << +/W 0 +>> +/Dest (cb7-9) +>> +endobj +87 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 544.561281 99.933071 525.811281 ] +/BS << +/W 0 +>> +/Dest (cb7-10) +>> +endobj +88 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 525.614260 99.933071 506.864260 ] +/BS << +/W 0 +>> +/Dest (cb7-11) +>> +endobj +89 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 506.667238 99.933071 487.917238 ] +/BS << +/W 0 +>> +/Dest (cb7-12) +>> +endobj +90 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 487.720217 99.933071 468.970217 ] +/BS << +/W 0 +>> +/Dest (cb7-13) +>> +endobj +91 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 468.773195 99.933071 450.023195 ] +/BS << +/W 0 +>> +/Dest (cb7-14) +>> +endobj +92 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 370.326174 101.433071 351.576174 ] +/BS << +/W 0 +>> +/Dest (cb8-1) +>> +endobj +93 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 351.379152 99.933071 332.629152 ] +/BS << +/W 0 +>> +/Dest (cb8-2) +>> +endobj +94 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 332.432131 99.933071 313.682131 ] +/BS << +/W 0 +>> +/Dest (cb8-3) +>> +endobj +95 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 294.538088 99.933071 275.788088 ] +/BS << +/W 0 +>> +/Dest (cb8-4) +>> +endobj +96 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 275.591066 99.933071 256.841066 ] +/BS << +/W 0 +>> +/Dest (cb8-5) +>> +endobj +97 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 298.350419 124.541066 391.740800 109.541066 ] +/BS << +/W 0 +>> +/A << +/Type /Action +/S /URI +/URI (file:///doc_tools/%7B%7B%3Cbaseurl%3E%7D%7D/rke/latest/en/installation/) +>> +>> +endobj +98 0 obj +<< +/Filter /FlateDecode +/Length 5338 +>> +stream +x]]$;r}_Q WW/0 6 |0vfc}P>232kN3ݕ!)8 +)*sW"R1UJEI MQyCׯ,pQm ~.쯗??Ks*`3'S1ŨW1Z`n1g) fHIfPN^Jdmp&ea!) f죐d40Vk1 h"g4U@)RYLH3JiΩx#eBIJ#6R-i) KQJ 4bStRZ-[$ҲS>'"1-#MiV{sZJ3Fy`Ӊi(tRB-A,ﴕl!Hi bJZnO '5LAW4IKȳ1>c&rm :y2įcO.F>]FxghlzC!$0~z/_P/b4&_3ncⳝ|$>Ϻp\S +sY|hmY?}.mv"mQn?i6oڮ !iN ^Z>4vyu@Os+~?Ϙq" E^$LBį083Ɂ_ +3n ,!O빝 >OmZۺ̟+ o_ e֍:ͻn#a"( jY +:V>SG0aaqnٟۈ6@>v]?#Z!it>4yJy"]yxy" xOK #Z8썼 + ̈ҥ~ᔧUhM7-ӧ.,T9UqYf+kԙ 2lH;6i2eua,{@/19LxQ9]a79<xa DnȾ4?ʼ ǃVN<5'u;c_}ᔺ?~YupIZ6ʼgݕ}i/'yj-*wIX\к) xq9.Đ;)4aEoU.>虢_Tonszޞƕi5<< 1lX+8F\ qZEe4;{Y:?/yQ9nk:+i1YN;u +dVrٴzǻYp) P(`9&o2O(v.w6AkJƮ4 :VC0 Őg0X.l˖R<;kcj[Q(g"9&D!,;tD,{`;o[So}L?/ N;`a@|+s'u#dG,x'<ums]bs?'_7a1A;s*OcoZ:5%ygpz[5> ͏5Y/ihe_?ҙ_AfegJWtq6ż(wվG3N;VŬҲN~a]w=c8<1Yu9tjp:*tB\n)TiztFD}uveH]U|zDw{fC +V+/%AX[^~3 (3=~ +GSΓ:"-{hQɂд$}͡JN'=Z1uO @>hs1Ѻ90cKi5 7&G;<weE%VT<M } 03KIjI^PPW|zUfa$ʻ!Cl`č ߅X*o4i}ykɼ<-+)CɋYT^ilm8^F}Mb L#Z7)p:r.3txJ~x?8PtH3@4^qNFпǀ|#2y%G@<<@l$ƇoE%?ćam8A7gQ[Pv9^D`7ޢNܬJ +z< 7pZ*tV('o'uIsА{O[Vq\2&/a:b#3AZ\B,e}Rye#ݷ֢R-H}Ix}jQ- + 2BgݞSyy~cV0ZTR`zr_v+~8-B7g 4@pBTss:D]hQI jǧ@)!h?B[{Jv(/G(N:4ʚj> @h{rmYpV1jyUc^ Z*Lrr; :+/m  )!HmԶ!EQl?<<pZT2p}N}nv}f>ivVjƋ2􀝨1O+O<")weE%jk뭮灴[Ǟ[ +w"\(n\9EgZ'ul|ԃ2~Gj*C3P#T'?uRK6>f2>ytd?OA07Ev*GyEJ̑46q{պUԢjZdn0*79[P'AtŸt-Й-}5>} +$jr'n1\o=[UsmnJKYiYѮ~\@k؍vCG!\(on(xҘjS5U&e) 5HW¾F塱I-QHQv c{fL-wkthARC#451SXmƕPW#& ʯ q7WUwиuㆿ0_2ZO.Ս +V 솁kWom۷d҈{R9;8_3FGwSܧqqB`Oj.eiסH\q>#절P\nLXIkzƮG&Fiԇٱ*+w1dAƋ`2cya`wlrFM:Jۍ,u2,/ۘʤ:ooV|ϵ F-hq1bt6gcTeJ@2]eXm +ZmrIlb.~ֈ~앰EK@~sw{K&AF&UٶnZ]Gp =2]or~}gtKOY0 j2|6e*[BY4,i> +endobj +100 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 289.074296 784.370079 316.109452 769.370079 ] +/BS << +/W 0 +>> +/A << +/Type /Action +/S /URI +/URI (file:///doc_tools/%7B%7B%3Cbaseurl%3E%7D%7D/rke/latest/en/config-options/nodes/) +>> +>> +endobj +101 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 650.584475 101.433071 631.834475 ] +/BS << +/W 0 +>> +/Dest (cb9-1) +>> +endobj +102 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 612.690432 99.933071 593.940432 ] +/BS << +/W 0 +>> +/Dest (cb9-2) +>> +endobj +103 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 574.796389 99.933071 556.046389 ] +/BS << +/W 0 +>> +/Dest (cb9-3) +>> +endobj +104 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 555.849367 99.933071 537.099367 ] +/BS << +/W 0 +>> +/Dest (cb9-4) +>> +endobj +105 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 536.902346 99.933071 518.152346 ] +/BS << +/W 0 +>> +/Dest (cb9-5) +>> +endobj +106 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 499.008303 99.933071 480.258303 ] +/BS << +/W 0 +>> +/Dest (cb9-6) +>> +endobj +107 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 480.061281 99.933071 461.311281 ] +/BS << +/W 0 +>> +/Dest (cb9-7) +>> +endobj +108 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 461.114260 99.933071 442.364260 ] +/BS << +/W 0 +>> +/Dest (cb9-8) +>> +endobj +109 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 442.167238 99.933071 423.417238 ] +/BS << +/W 0 +>> +/Dest (cb9-9) +>> +endobj +110 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 423.220217 99.933071 404.470217 ] +/BS << +/W 0 +>> +/Dest (cb9-10) +>> +endobj +111 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 404.273195 99.933071 385.523195 ] +/BS << +/W 0 +>> +/Dest (cb9-11) +>> +endobj +112 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 385.326174 99.933071 366.576174 ] +/BS << +/W 0 +>> +/Dest (cb9-12) +>> +endobj +113 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 366.379152 99.933071 347.629152 ] +/BS << +/W 0 +>> +/Dest (cb9-13) +>> +endobj +114 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 347.432131 99.933071 328.682131 ] +/BS << +/W 0 +>> +/Dest (cb9-14) +>> +endobj +115 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 328.485109 99.933071 309.735109 ] +/BS << +/W 0 +>> +/Dest (cb9-15) +>> +endobj +116 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 309.538088 99.933071 290.788088 ] +/BS << +/W 0 +>> +/Dest (cb9-16) +>> +endobj +117 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 290.591066 99.933071 271.841066 ] +/BS << +/W 0 +>> +/Dest (cb9-17) +>> +endobj +118 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 271.644045 99.933071 252.894045 ] +/BS << +/W 0 +>> +/Dest (cb9-18) +>> +endobj +119 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 252.697023 99.933071 233.947023 ] +/BS << +/W 0 +>> +/Dest (cb9-19) +>> +endobj +120 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 233.750002 99.933071 215.000002 ] +/BS << +/W 0 +>> +/Dest (cb9-20) +>> +endobj +121 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 214.802980 99.933071 196.052980 ] +/BS << +/W 0 +>> +/Dest (cb9-21) +>> +endobj +122 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 195.855959 99.933071 177.105959 ] +/BS << +/W 0 +>> +/Dest (cb9-22) +>> +endobj +123 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 176.908937 99.933071 158.158937 ] +/BS << +/W 0 +>> +/Dest (cb9-23) +>> +endobj +124 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 157.961916 99.933071 139.211916 ] +/BS << +/W 0 +>> +/Dest (cb9-24) +>> +endobj +125 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 139.014895 99.933071 120.264895 ] +/BS << +/W 0 +>> +/Dest (cb9-25) +>> +endobj +126 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 120.067873 99.933071 101.317873 ] +/BS << +/W 0 +>> +/Dest (cb9-26) +>> +endobj +127 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 101.120852 99.933071 82.370852 ] +/BS << +/W 0 +>> +/Dest (cb9-27) +>> +endobj +128 0 obj +<< +/Filter /FlateDecode +/Length 5039 +>> +stream +x]]}_ρUQ,$0$!p7C~N-YnU۳a^Wtt$˶9kDFc'3XL?_g)*orH \T1{no﵍9[sƟUTL1j/2T8A彊 QĜ%,7('/YUq*  I*aha֢h"g*уFmu:sNHIU'IGgm$/iI_ȥ(|qiQ_lNEb:Y%t%Q_tFe|ND.CSLj|rNK:c6ÈB'(,| gQ_V9B+Ŕjyt;팤H%$ +RKeK %J ؔ$+ +9J<ഁ/ueY8_,9 ؖ t68-i$.p \%.p`\I\@RsSp8J\@Rw-qw/`GLrx%. . qlJZڤIڈ txz\k+qw- q?Z/+q=tI\)D5O`+eRxI>Qhqaܢ !K\TN}& +J\!RK:YK\2H$.@|SF.+SwAKVDaDE  @J# w޳s&=d%.@^0*m zXҁwY)qA,I8[ B"c C'± j<(K$qk]ND _D _2H\@e[E|A(qLL$qA1 Vu :%.Bs)0%.+N$. f"qBJ3It3Fd A Me-q"4hjIGIDAq(q f`JQ_@x%.MLsD|q$.Ht3I\|J\l|3J\5|A'qAAq,qh:%.&aiIKo& Әob'7G" ,[p`'I$qAIo"7\$L8|4 ^3$hPGl 4*ks$qnbfj +ⲁ"OU"bf|Â9R)v]?N{O_~5wxby،]t6j!|ƆyqҀE7v6/f2 yV[!HcO ;MSTĹ@@)-s'^M¿p0oDsBZ}~?~rN?<96уVui_OE)̿iEv{;4s<ߧ; 1<Mi|b4EC)`^.\4}i~N݇5ΒWyi@S|1xx"AcuΏ*c|N8lAA%|$ ɝHLA&f: ;˧_7z^u6 ۦ{ӄ J|)rT&%Bјq@l3zѪ /οTv&]|?Wmߝ"af\LtۜSuw2&[9"YWlUJT4YO)72]ۀkYnɸz{=%enS;Fo2ژagyB+\ї4otSm-[@lxob$ 5PSmm1_iJPI,t>lo- 6- Zxqtaa8|$xJ +x_d(u>mRg=Dgd n",i- I!WqUWqw]2ދ=ZU!R{ |ƤL][9 xW2 c6 yF.UǺZ\c,W-w7tۼ92z73)x-n*7mȊ229x-dsJoIګ2o@pCT9PIm6G\ڴӮf9p=D_ U%pBj$P3PA`|$Γz=}[0uu4%oc6ǁn"2WBwUɴ DM/y+-P.nSdB26r%`) T-V-6!h^JS4==a Kj~u]W" }vzU&E[k;B+wS \wbm*~Po@zsSj4,}`ܞ+/}]{{}PφK#c`iKcCho UlfN^G6eQhjfoo5߿S +'RԨv[Ax8`M>Yz/ǃ𪒒7Jl?5[*ӘM5,x7(26va:kqߞD4#+}mP{}A=ΚAZƃq &̪؂<W wBp'ao?z$lHcaǑt80, νr麒R8C0+P wk>nnس|[[ퟀ47؃6ǁ4Fg~ !dpgZ"}iN ZOf6A@!h~:zzη J_u{}y`o/ &`<o mAǏo Ul'偽(#6ǁgޢ1["[tcdwۻ7Lv{a38-~A!q+7c+gy ``oIe7{p6Zm9+7+y3h}]8)?"9N;~fyͺnPeO,lޟ=Mo]Z5z(xk?]8m]hC|Vdl1G*xGHrܯ;ߦ]nl4+%8|>{[[M] v2_KaY>UGB(qx UG .*݆c3wguL&`AsK!fql^.)+ &~_Wooc8HĖA.x|Mӵh-mӵY3M÷Ȳc;:o_>V7׵k?;Bw.ez}Kj/p{$8LGol +1r˺o=*GV['g36gAs|ںWm>z>!C~/車mO) ^ U%̻zvzg[P@wΏh0^m"%zZ"=w{}[Y淞cV8 %E67)ln +u!8ş;gۈ\@~^pJ?^lvڴ4ߒ˱*w$w0aLi3creS箹>f*Sžy)|`$ЋsAǟk)"͟ Ssd*|H젘󯺞@.ྨmI畞Η_/W1QvrXV,aL/ט_ؤ|+>!\NHm儋ᦘwr,B uS"kCX]o4_t#jݭMWʩJZ]eq*W/%kP'At9 t955VU +˥ r|XK4_\YRmxA6,fhW~]yslkNx9P/\}^+ty8/_Lv CK*|2L,GK%<6xH> +endobj +130 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 799.173057 99.933071 780.423057 ] +/BS << +/W 0 +>> +/Dest (cb9-28) +>> +endobj +131 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 780.226036 99.933071 761.476036 ] +/BS << +/W 0 +>> +/Dest (cb9-29) +>> +endobj +132 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 761.279014 99.933071 742.529014 ] +/BS << +/W 0 +>> +/Dest (cb9-30) +>> +endobj +133 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 742.331993 99.933071 723.581993 ] +/BS << +/W 0 +>> +/Dest (cb9-31) +>> +endobj +134 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 723.384971 99.933071 704.634971 ] +/BS << +/W 0 +>> +/Dest (cb9-32) +>> +endobj +135 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 704.437950 99.933071 685.687950 ] +/BS << +/W 0 +>> +/Dest (cb9-33) +>> +endobj +136 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 685.490928 99.933071 666.740928 ] +/BS << +/W 0 +>> +/Dest (cb9-34) +>> +endobj +137 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 666.543907 99.933071 647.793907 ] +/BS << +/W 0 +>> +/Dest (cb9-35) +>> +endobj +138 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 647.596885 99.933071 628.846885 ] +/BS << +/W 0 +>> +/Dest (cb9-36) +>> +endobj +139 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 628.649864 99.933071 609.899864 ] +/BS << +/W 0 +>> +/Dest (cb9-37) +>> +endobj +140 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 609.702842 99.933071 590.952842 ] +/BS << +/W 0 +>> +/Dest (cb9-38) +>> +endobj +141 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 590.755821 99.933071 572.005821 ] +/BS << +/W 0 +>> +/Dest (cb9-39) +>> +endobj +142 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 571.808799 99.933071 553.058799 ] +/BS << +/W 0 +>> +/Dest (cb9-40) +>> +endobj +143 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 552.861778 99.933071 534.111778 ] +/BS << +/W 0 +>> +/Dest (cb9-41) +>> +endobj +144 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 533.914756 99.933071 515.164756 ] +/BS << +/W 0 +>> +/Dest (cb9-42) +>> +endobj +145 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 514.967735 99.933071 496.217735 ] +/BS << +/W 0 +>> +/Dest (cb9-43) +>> +endobj +146 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 496.020714 99.933071 477.270714 ] +/BS << +/W 0 +>> +/Dest (cb9-44) +>> +endobj +147 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 477.073692 99.933071 458.323692 ] +/BS << +/W 0 +>> +/Dest (cb9-45) +>> +endobj +148 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 458.126671 99.933071 439.376671 ] +/BS << +/W 0 +>> +/Dest (cb9-46) +>> +endobj +149 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 439.179649 99.933071 420.429649 ] +/BS << +/W 0 +>> +/Dest (cb9-47) +>> +endobj +150 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 420.232628 99.933071 401.482628 ] +/BS << +/W 0 +>> +/Dest (cb9-48) +>> +endobj +151 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 401.285606 99.933071 382.535606 ] +/BS << +/W 0 +>> +/Dest (cb9-49) +>> +endobj +152 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 382.338585 99.933071 363.588585 ] +/BS << +/W 0 +>> +/Dest (cb9-50) +>> +endobj +153 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 363.391563 99.933071 344.641563 ] +/BS << +/W 0 +>> +/Dest (cb9-51) +>> +endobj +154 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 344.444542 99.933071 325.694542 ] +/BS << +/W 0 +>> +/Dest (cb9-52) +>> +endobj +155 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 325.497520 99.933071 306.747520 ] +/BS << +/W 0 +>> +/Dest (cb9-53) +>> +endobj +156 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 306.550499 99.933071 287.800499 ] +/BS << +/W 0 +>> +/Dest (cb9-54) +>> +endobj +157 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 287.603477 99.933071 268.853477 ] +/BS << +/W 0 +>> +/Dest (cb9-55) +>> +endobj +158 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 173.921348 99.933071 155.171348 ] +/BS << +/W 0 +>> +/Dest (cb9-56) +>> +endobj +159 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 154.974327 99.933071 136.224327 ] +/BS << +/W 0 +>> +/Dest (cb9-57) +>> +endobj +160 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 136.027305 99.933071 117.277305 ] +/BS << +/W 0 +>> +/Dest (cb9-58) +>> +endobj +161 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 117.080284 99.933071 98.330284 ] +/BS << +/W 0 +>> +/Dest (cb9-59) +>> +endobj +162 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 98.133262 99.933071 79.383262 ] +/BS << +/W 0 +>> +/Dest (cb9-60) +>> +endobj +163 0 obj +<< +/Filter /FlateDecode +/Length 4884 +>> +stream +x]]8}_/haahh؁bf؇;U3,apZ̔oUVt +KGG!9lşL%E>|dM +M%V/YN&s8,J&S[TpH3s,辭9bte{iq?qu! te1:ڙIȓp-6D*8o1ݖs.c!zRp*mT8m{\i5ݏwOￗܭݟԁº :4n6;@W&;A1jqex<vmsn #cZ \|c&o304KaOܪ9-ggT,򊷆0c..XGDL2֑5nGpvk.;A-l:iһ9 u%; +eԕnX>ʛ_?_% uo!t/.H`hEb^OxKtPh͐c6/Vo>+fY'Nki +O; 0PjtnDD#x.;nslӹInQ3iU9cW=ܰ|sŗ٭CdYֲ5,[l"/)'h]~ +y>0˃.`~J]8BP;l"vKvq)0,o-+ +]pWWGF>#ZE3+< ͸~#)eeuknr'&1#P q }nc6ǁѕ[ YAQ3\l_||=ñT˹s]lky? +{>qZva8p 3pVݵ N]}nZ[4&?2_KeY/:jp[|9lIKyl`)e%y*JvUz3 Mvkk(;2X쓤p{tm8ɛ|ᙀ[U2w9N1*=ز X]dWG+Yp +bc6ǁX@Q[UE;wwÞwkoUZqvS^7fsIbgoU 'JMw@Xn (1ߑ7hs 3SA֕Vo[/m%mA/|)y.Ly&V#KjӞD[}YR[ \o4Kj0@,V#bU%U֋=YR[}YR[{+ݢw7hs<S|fRͺ/zOXcKhgXVCiӞxk][-WVi6\R9H<%o&HW|.u{nKodYޖ H*?rzk=FbE(/{mU.1oA[ײʊYuk1)%=*y)b>G p.jՅs˫y +75՛_qG0LC={/pϟYL\d/32r""88)72+^l\;[16NFb$ ڼnjwa9}VJʇz$Yaj`.1?EQ Zv˪Zz/"F"v.I +\`S?}W# ]HzX2X"+t|+!V!\N}儋C$/°ڮE,_e}YZC\u_u= ҵUʷjzd9U4ԒC"z9)l=5(^/\[R)Zn>[h=Z+ )WdS$ƈs-mp7—<BRJ+lԝ]Hԗ_U&_/TTgAJ2Mn^rm8wJ }{lBtkkNJuOHZX(O^hnוx3l,wƝ8wAo;o*|&6 sd{fN8RX N:Z;I^.Lq_N> +endobj +165 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 799.173057 99.933071 780.423057 ] +/BS << +/W 0 +>> +/Dest (cb9-61) +>> +endobj +166 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 780.226036 99.933071 761.476036 ] +/BS << +/W 0 +>> +/Dest (cb9-62) +>> +endobj +167 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 761.279014 99.933071 742.529014 ] +/BS << +/W 0 +>> +/Dest (cb9-63) +>> +endobj +168 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 742.331993 99.933071 723.581993 ] +/BS << +/W 0 +>> +/Dest (cb9-64) +>> +endobj +169 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 723.384971 99.933071 704.634971 ] +/BS << +/W 0 +>> +/Dest (cb9-65) +>> +endobj +170 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 704.437950 99.933071 685.687950 ] +/BS << +/W 0 +>> +/Dest (cb9-66) +>> +endobj +171 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 685.490928 99.933071 666.740928 ] +/BS << +/W 0 +>> +/Dest (cb9-67) +>> +endobj +172 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 571.808799 99.933071 553.058799 ] +/BS << +/W 0 +>> +/Dest (cb9-68) +>> +endobj +173 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 552.861778 99.933071 534.111778 ] +/BS << +/W 0 +>> +/Dest (cb9-69) +>> +endobj +174 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 533.914756 99.933071 515.164756 ] +/BS << +/W 0 +>> +/Dest (cb9-70) +>> +endobj +175 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 514.967735 99.933071 496.217735 ] +/BS << +/W 0 +>> +/Dest (cb9-71) +>> +endobj +176 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 496.020714 99.933071 477.270714 ] +/BS << +/W 0 +>> +/Dest (cb9-72) +>> +endobj +177 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 477.073692 99.933071 458.323692 ] +/BS << +/W 0 +>> +/Dest (cb9-73) +>> +endobj +178 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 458.126671 99.933071 439.376671 ] +/BS << +/W 0 +>> +/Dest (cb9-74) +>> +endobj +179 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 439.179649 99.933071 420.429649 ] +/BS << +/W 0 +>> +/Dest (cb9-75) +>> +endobj +180 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 420.232628 99.933071 401.482628 ] +/BS << +/W 0 +>> +/Dest (cb9-76) +>> +endobj +181 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 401.285606 99.933071 382.535606 ] +/BS << +/W 0 +>> +/Dest (cb9-77) +>> +endobj +182 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 382.338585 99.933071 363.588585 ] +/BS << +/W 0 +>> +/Dest (cb9-78) +>> +endobj +183 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 363.391563 99.933071 344.641563 ] +/BS << +/W 0 +>> +/Dest (cb9-79) +>> +endobj +184 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 249.709434 99.933071 230.959434 ] +/BS << +/W 0 +>> +/Dest (cb9-80) +>> +endobj +185 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 230.762413 99.933071 212.012413 ] +/BS << +/W 0 +>> +/Dest (cb9-81) +>> +endobj +186 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 211.815391 99.933071 193.065391 ] +/BS << +/W 0 +>> +/Dest (cb9-82) +>> +endobj +187 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 192.868370 99.933071 174.118370 ] +/BS << +/W 0 +>> +/Dest (cb9-83) +>> +endobj +188 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 173.921348 99.933071 155.171348 ] +/BS << +/W 0 +>> +/Dest (cb9-84) +>> +endobj +189 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 154.974327 99.933071 136.224327 ] +/BS << +/W 0 +>> +/Dest (cb9-85) +>> +endobj +190 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 136.027305 99.933071 117.277305 ] +/BS << +/W 0 +>> +/Dest (cb9-86) +>> +endobj +191 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 117.080284 99.933071 98.330284 ] +/BS << +/W 0 +>> +/Dest (cb9-87) +>> +endobj +192 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 98.133262 99.933071 79.383262 ] +/BS << +/W 0 +>> +/Dest (cb9-88) +>> +endobj +193 0 obj +<< +/Filter /FlateDecode +/Length 5023 +>> +stream +x]ߋ$~ +K/X` 3 w=kXEVJ̌T@ݕ!}T2;sI^~Xſb* ~R:Wb_@ɤ|׷ ~m.okw])b2)dRTPd)E!|JQtXV1Bb1A >+Eb'G1|rA WBR[AL^)sD zʼ31_Բ2VFdbbVVL1d!K(iel2”V! Ejd9V ,CXWV ieL(Y-+(9}Ld2LmZ%Ťym(jd dVF_+NJ9+&KN+ jd55 2K L3g |.S" | cIu(N%pԐda\ Pa CTаx& h˰ p˶8 xE  1ĥY”%`tOPB`8G K쬆 %ZY` L0pM0pkXa԰b|VlY%9H0p܂湁>Xaw=a  w!A V{yҰDÂuO@+rPZVKqebhXpTaA>x5,]GZpb5,] Aq4,3tղblYÂ,y @,Ӱ`l,P "phX6iXeװq 0m9`XʀAZpT4,]q$5,Qװ f6%aA,eԲhp4,HR 'ְ)TC^Â! B }dCDL.aTMNk,% RbW4,@.e2&x#ᒆe5,H`hXWdGd9Ȃ [D(sNÂ! A 2հ$Z+c+yfְ s4*jXiLHAЏ$A$?H6ekXt3kX3dkX۔|3iXP,dAħaAHa`ejXP\FYK7,Mxʼ-%L,!ZY4eoY|a Jof f DD. d7\ ȤwF p|[oH,gVW ,7K o5 I]4,sTW VQ謺l`x է~*|fNrӂݷcFw_|M+eGIL_nc(ڙȓ=<"D*8/pp>ȐXdvf;_d +4GJ_L?ސZ 0cO O|HbZT`'&^˫t4жNnxsz{+G ~??1u)V}H|k*vM3uvV\ϵǷHa;r?0  ߱[n9oo珟֖a0!occyG`A߱4bH6cWmN621XOQ4U'CV}S7zי̮c f=bp:@a5~=˒+bs="i ˊ +-;ܺ! PJo& {?21IvuPTG yLiiN&W`xgŘI +RC~fu8S/Ol"O(|<41˪՞{hݩ[)֝q?dtr<qjs?ix4ҙîg1-b*w ,&$rKc9)%F})vxFgsqJ[BqI7lbQG$yNc.leח>\֍HEx[]](E/+{ɞYs?<bxk٠3|#o.Z"jzxO6s}{^۩'3ӞI:Mdy,0RqG&9s %@INɧT[K$۸kx{- ^my&X⑺xΣ.2G"Ȫ?~<*,𾯁𹅿{T yIu?GSi4,ǧm$n7ɄƳ.޷yF/a{l=6J5Q1sÑW2ŏTntUs'5>^}_Ϥ{xNNC>f ZOG4r# BD:: C oYPf(Ɗ]@,/P[:w7s+Cϊt"s(v<؁?"hq-Oߗ*Gbn߸|nPcumkgDE^奒,i(Zy/r:CY#`{Xn=dQjHUAwlі' y!uFu-s'ȢAlA\R DHbv:R9wf0 R8իF1]φ0N0n?MYkr_tP"sS:ZJ׾_s[#|G xO5~0YN]HѺۆ!D(9ay$9or=|Ksvb*~l0Lx,cCwJY5C?D0>Q('A(?.nG}D63c=\o keZx1.h1p0r+w f{4Wb\Lr(QgI6qjl]Se:*7wt(i5A(;CͧQWˋ9 <,`4]9RGvAJ DO2sLڹęqNпR1Ib?t|T{%Y.]k +8r0]t78h0E釜=yӟa@ܮĺLRlF<\O=s^;Q +;ՙǝ=&ꄮx#nK:uy6NFr<19g*fKi@5"Z$+4\I1^dܣX[DՕ-" +n8-yb'S5);,q +n?T69"F=+[ +r9ȍ, W\*4}ݥ,[bmBj]>sйNO:_/薶7*צzh U5ԚM!^/ +[hwZSo\ [R)׋Zon-4MV+/"b+ifj戥qMk?"vKHJ` z"u6r,ԔPajV!+oPUgน,-WK'27y=H==6am`$픚U mVn:z*U<kkVau|VD[@w 94S~[9C΢=vrl֓SӸC+url%B|Zgq }u6tei</i$'QGGgן7?mC ߰uE=|Q$7Ub6s n^n`5]CPJ{bXư#fQ72u3-V,vI9NJvp2Z\kWRuq(e1o]el%dIz{U]ta\RƋ훮.*fwZV(B{z}gcBmިo=ݘPT.{wusZL%Eo5/lswʕ:Jk-*SLwQJ-XG:Ziz]]iz-Q Boi;ldVhnT^jRu^EDva ¿̝f@UO+./_#G.!m>s45JF]{: ;>uPP l_@YY`Vp/yÿy!]&I;I,_EStzas81 +endstream +endobj +194 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 193 0 R +/Resources 4 0 R +/Annots [ 195 0 R 196 0 R 197 0 R 198 0 R 199 0 R 200 0 R 201 0 R 202 0 R 203 0 R 204 0 R 205 0 R 206 0 R 207 0 R 208 0 R 209 0 R 210 0 R 211 0 R 212 0 R 213 0 R 214 0 R 215 0 R 216 0 R 217 0 R 218 0 R 219 0 R 220 0 R 221 0 R 222 0 R 223 0 R 224 0 R 225 0 R 226 0 R 227 0 R 228 0 R 229 0 R 230 0 R ] +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +195 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 799.173057 99.933071 780.423057 ] +/BS << +/W 0 +>> +/Dest (cb9-89) +>> +endobj +196 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 780.226036 99.933071 761.476036 ] +/BS << +/W 0 +>> +/Dest (cb9-90) +>> +endobj +197 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 761.279014 99.933071 742.529014 ] +/BS << +/W 0 +>> +/Dest (cb9-91) +>> +endobj +198 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 742.331993 99.933071 723.581993 ] +/BS << +/W 0 +>> +/Dest (cb9-92) +>> +endobj +199 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 723.384971 99.933071 704.634971 ] +/BS << +/W 0 +>> +/Dest (cb9-93) +>> +endobj +200 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 704.437950 99.933071 685.687950 ] +/BS << +/W 0 +>> +/Dest (cb9-94) +>> +endobj +201 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 685.490928 99.933071 666.740928 ] +/BS << +/W 0 +>> +/Dest (cb9-95) +>> +endobj +202 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 666.543907 99.933071 647.793907 ] +/BS << +/W 0 +>> +/Dest (cb9-96) +>> +endobj +203 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 647.596885 99.933071 628.846885 ] +/BS << +/W 0 +>> +/Dest (cb9-97) +>> +endobj +204 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 628.649864 99.933071 609.899864 ] +/BS << +/W 0 +>> +/Dest (cb9-98) +>> +endobj +205 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 609.702842 99.933071 590.952842 ] +/BS << +/W 0 +>> +/Dest (cb9-99) +>> +endobj +206 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 590.755821 99.933071 572.005821 ] +/BS << +/W 0 +>> +/Dest (cb9-100) +>> +endobj +207 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 571.808799 99.933071 553.058799 ] +/BS << +/W 0 +>> +/Dest (cb9-101) +>> +endobj +208 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 552.861778 99.933071 534.111778 ] +/BS << +/W 0 +>> +/Dest (cb9-102) +>> +endobj +209 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 533.914756 99.933071 515.164756 ] +/BS << +/W 0 +>> +/Dest (cb9-103) +>> +endobj +210 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 514.967735 99.933071 496.217735 ] +/BS << +/W 0 +>> +/Dest (cb9-104) +>> +endobj +211 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 496.020714 99.933071 477.270714 ] +/BS << +/W 0 +>> +/Dest (cb9-105) +>> +endobj +212 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 477.073692 99.933071 458.323692 ] +/BS << +/W 0 +>> +/Dest (cb9-106) +>> +endobj +213 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 458.126671 99.933071 439.376671 ] +/BS << +/W 0 +>> +/Dest (cb9-107) +>> +endobj +214 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 439.179649 99.933071 420.429649 ] +/BS << +/W 0 +>> +/Dest (cb9-108) +>> +endobj +215 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 420.232628 99.933071 401.482628 ] +/BS << +/W 0 +>> +/Dest (cb9-109) +>> +endobj +216 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 401.285606 99.933071 382.535606 ] +/BS << +/W 0 +>> +/Dest (cb9-110) +>> +endobj +217 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 344.444542 99.933071 325.694542 ] +/BS << +/W 0 +>> +/Dest (cb9-111) +>> +endobj +218 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 325.497520 99.933071 306.747520 ] +/BS << +/W 0 +>> +/Dest (cb9-112) +>> +endobj +219 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 306.550499 99.933071 287.800499 ] +/BS << +/W 0 +>> +/Dest (cb9-113) +>> +endobj +220 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 287.603477 99.933071 268.853477 ] +/BS << +/W 0 +>> +/Dest (cb9-114) +>> +endobj +221 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 268.656456 99.933071 249.906456 ] +/BS << +/W 0 +>> +/Dest (cb9-115) +>> +endobj +222 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 249.709434 99.933071 230.959434 ] +/BS << +/W 0 +>> +/Dest (cb9-116) +>> +endobj +223 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 230.762413 99.933071 212.012413 ] +/BS << +/W 0 +>> +/Dest (cb9-117) +>> +endobj +224 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 211.815391 99.933071 193.065391 ] +/BS << +/W 0 +>> +/Dest (cb9-118) +>> +endobj +225 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 192.868370 99.933071 174.118370 ] +/BS << +/W 0 +>> +/Dest (cb9-119) +>> +endobj +226 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 173.921348 99.933071 155.171348 ] +/BS << +/W 0 +>> +/Dest (cb9-120) +>> +endobj +227 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 154.974327 99.933071 136.224327 ] +/BS << +/W 0 +>> +/Dest (cb9-121) +>> +endobj +228 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 136.027305 99.933071 117.277305 ] +/BS << +/W 0 +>> +/Dest (cb9-122) +>> +endobj +229 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 117.080284 99.933071 98.330284 ] +/BS << +/W 0 +>> +/Dest (cb9-123) +>> +endobj +230 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 98.133262 99.933071 79.383262 ] +/BS << +/W 0 +>> +/Dest (cb9-124) +>> +endobj +231 0 obj +<< +/Filter /FlateDecode +/Length 4449 +>> +stream +x]ߋ$~bU/~ÑB>?_Ꞛ^Inٛ*I]]%}U%t_rԜɻ=M%V;x&/E'%pvƐ6cӟ~s5_~P%g V]!&kX#f rCj bH`YbLb ՘ yA{AG%(=Y@.Q٥,UR dՓŋ%xEd '&ej +dA*&UvW%k^-l0` ^,j Kd,CNDΥ5fd"d !Ƀ,0!AANq€PIb@(Il@I^ C(B&"&p 8@a h+-0V|% x+ rIbarѕ e` p0Pc +0`$ B2Ƀ,Sa 䳅0 +0YXB) `d "}@AB w! {[X\@XbaAR34Q‚ܥ(CƸT-,]8* w`G D.1<R‚E-,@wp7G]Uk)D.`)XX/|БC-,H]jaAlaAzAc `a0l1`x]Oz n‚! +",,HՃ%haA D@Ƀ,֮r-Y8 r,,,@D*!6ȂbY)ZX$sXXcB>Z\ɀkWlaAR`[Xw.YX1# WȂba\+#Ȃ DYXPdA4haA YfC#-x3EP d@JY>lG Dc Gf|}SlaA'aq𼷰RAK7=MxƸ4T,!Ƀ,1nI7N/AP‚7ћSElNwc2Y<."da.o"r{,qsz#|9eVٜ3_͑I̲$snL洁"+L>ST !gY1,}9b|e~O_=L3eY&QH07M Mn"$!|M"EH,Γn*$\1m=/̄`fz2L`*N/Vr/og9 risUBRHcR\b'^/¿{w0o헃~廯?>[ ^I~C77\h=~=]>c=_߭\u#Uy>]m"C6:o ->$_+4@ %?0:u$,R賙u6D*d4tg qL; nvƯ Ip;GQ3FJ71\NAcsQtף@MoaFngØLosh(I*e_㘅u'q׎C\s47,X0[})TJGXIJ>zf_=*4-w6[}ڪK-i +l쾫}uΈ1#Bk16Kc?^,\ Yi>߬iYmmݖz/Ȧ+[{{-snQ~-4"|mqWEFx]iJ)(qjF~`}Ucr)}K qi>o]U LU1v9j7}b^W7,]&Av$4{~sd]7s-z=ྍwU9Ӣ;Kf*]9qa99AY[Qfc,µ,<,X"g9+a>Hp +4+,bHDgܗ5fz%[Kʬfg.e=cS^Ns)ѐ5Nep4X~+u|J9C45#ԢG9ݲY6*ziRoԜOzПNjO/?Ot"}F홗8D^ݐOzdmԆ]ݖ9!UӓqtFҾzADDd}zh&)|(15A\R!xMʆvEUG[ِs#>p[RMٚj xvjl%A = 5:@v!֑Az:bTR6/,]KWAt4S=뚜HZ~O'AtE~>l VغHN.~/ 5* F TNU䙾.g-WMtlvZI#=uK$y);r}voWVAe=&T5zd#7!p{W~^9c%{.Աy$nqݡ7dK֮_Mr-7`Dn( <ǠgbaDom]x-BydzxIdLe)~<e`o%#(uHr]7wO-@fqؗKn۰eפ8, |ĕ]\:púHS;xz_FSk'n)"yoZWBzxj"6ڡSGj0v<]e9<c{ %GjYxSiMpobn2ZSR1BitҕGnx=Zű;RǒŠvR޲w1d$-_ū¿׌KlUL3T#<ᨅ2*/)CDR)׹?q=8;N}&:`njwjx)Y`_L`扜~5UԿogO4W'Ͽ2 +endstream +endobj +232 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 231 0 R +/Resources 4 0 R +/Annots [ 233 0 R 234 0 R 235 0 R 236 0 R 237 0 R 238 0 R 239 0 R 240 0 R 241 0 R 242 0 R 243 0 R 244 0 R 245 0 R 246 0 R 247 0 R 248 0 R 249 0 R 250 0 R 251 0 R 252 0 R 253 0 R 254 0 R 255 0 R 256 0 R 257 0 R 258 0 R 259 0 R 260 0 R 261 0 R 262 0 R 263 0 R 264 0 R 265 0 R 266 0 R 267 0 R 268 0 R ] +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +233 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 799.173057 99.933071 780.423057 ] +/BS << +/W 0 +>> +/Dest (cb9-125) +>> +endobj +234 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 780.226036 99.933071 761.476036 ] +/BS << +/W 0 +>> +/Dest (cb9-126) +>> +endobj +235 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 761.279014 99.933071 742.529014 ] +/BS << +/W 0 +>> +/Dest (cb9-127) +>> +endobj +236 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 742.331993 99.933071 723.581993 ] +/BS << +/W 0 +>> +/Dest (cb9-128) +>> +endobj +237 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 704.437950 99.933071 685.687950 ] +/BS << +/W 0 +>> +/Dest (cb9-129) +>> +endobj +238 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 685.490928 99.933071 666.740928 ] +/BS << +/W 0 +>> +/Dest (cb9-130) +>> +endobj +239 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 666.543907 99.933071 647.793907 ] +/BS << +/W 0 +>> +/Dest (cb9-131) +>> +endobj +240 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 647.596885 99.933071 628.846885 ] +/BS << +/W 0 +>> +/Dest (cb9-132) +>> +endobj +241 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 628.649864 99.933071 609.899864 ] +/BS << +/W 0 +>> +/Dest (cb9-133) +>> +endobj +242 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 609.702842 99.933071 590.952842 ] +/BS << +/W 0 +>> +/Dest (cb9-134) +>> +endobj +243 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 590.755821 99.933071 572.005821 ] +/BS << +/W 0 +>> +/Dest (cb9-135) +>> +endobj +244 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 571.808799 99.933071 553.058799 ] +/BS << +/W 0 +>> +/Dest (cb9-136) +>> +endobj +245 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 552.861778 99.933071 534.111778 ] +/BS << +/W 0 +>> +/Dest (cb9-137) +>> +endobj +246 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 533.914756 99.933071 515.164756 ] +/BS << +/W 0 +>> +/Dest (cb9-138) +>> +endobj +247 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 514.967735 99.933071 496.217735 ] +/BS << +/W 0 +>> +/Dest (cb9-139) +>> +endobj +248 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 477.073692 99.933071 458.323692 ] +/BS << +/W 0 +>> +/Dest (cb9-140) +>> +endobj +249 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 458.126671 99.933071 439.376671 ] +/BS << +/W 0 +>> +/Dest (cb9-141) +>> +endobj +250 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 439.179649 99.933071 420.429649 ] +/BS << +/W 0 +>> +/Dest (cb9-142) +>> +endobj +251 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 420.232628 99.933071 401.482628 ] +/BS << +/W 0 +>> +/Dest (cb9-143) +>> +endobj +252 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 401.285606 99.933071 382.535606 ] +/BS << +/W 0 +>> +/Dest (cb9-144) +>> +endobj +253 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 382.338585 99.933071 363.588585 ] +/BS << +/W 0 +>> +/Dest (cb9-145) +>> +endobj +254 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 363.391563 99.933071 344.641563 ] +/BS << +/W 0 +>> +/Dest (cb9-146) +>> +endobj +255 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 344.444542 99.933071 325.694542 ] +/BS << +/W 0 +>> +/Dest (cb9-147) +>> +endobj +256 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 325.497520 99.933071 306.747520 ] +/BS << +/W 0 +>> +/Dest (cb9-148) +>> +endobj +257 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 306.550499 99.933071 287.800499 ] +/BS << +/W 0 +>> +/Dest (cb9-149) +>> +endobj +258 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 287.603477 99.933071 268.853477 ] +/BS << +/W 0 +>> +/Dest (cb9-150) +>> +endobj +259 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 268.656456 99.933071 249.906456 ] +/BS << +/W 0 +>> +/Dest (cb9-151) +>> +endobj +260 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 249.709434 99.933071 230.959434 ] +/BS << +/W 0 +>> +/Dest (cb9-152) +>> +endobj +261 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 230.762413 99.933071 212.012413 ] +/BS << +/W 0 +>> +/Dest (cb9-153) +>> +endobj +262 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 211.815391 99.933071 193.065391 ] +/BS << +/W 0 +>> +/Dest (cb9-154) +>> +endobj +263 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 192.868370 99.933071 174.118370 ] +/BS << +/W 0 +>> +/Dest (cb9-155) +>> +endobj +264 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 173.921348 99.933071 155.171348 ] +/BS << +/W 0 +>> +/Dest (cb9-156) +>> +endobj +265 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 154.974327 99.933071 136.224327 ] +/BS << +/W 0 +>> +/Dest (cb9-157) +>> +endobj +266 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 136.027305 99.933071 117.277305 ] +/BS << +/W 0 +>> +/Dest (cb9-158) +>> +endobj +267 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 117.080284 99.933071 98.330284 ] +/BS << +/W 0 +>> +/Dest (cb9-159) +>> +endobj +268 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 98.133262 99.933071 79.383262 ] +/BS << +/W 0 +>> +/Dest (cb9-160) +>> +endobj +269 0 obj +<< +/Filter /FlateDecode +/Length 4154 +>> +stream +x]ߋ$ ~￢-ٖ!C `H!w^SuL{" 3%U*,ׄW%<~J:ťPg%PqX_c_~wN8FWk!,&W V2X)Rb+WbAT85@ ,` .YbbAPS1XC7y3S4x! gRc5yP#rpHUsNbK%fo ! e&D)d +dN5xYкA}x\l*x^\O. x!I&#b"d>UYh0-p"Q15w`dɃ,01C_ da@,ql@r,, +DK! bMs-`yȒ8[H@0 rAj w|&Pqoer$.:* wba0ޡG$n b$.da ,,@G5y*baA腝 + )XX0vqZX>YXkkW2yk:k^a{ߘ3"4eDQtA&"ӑq7qDe=t.{vk9xFGno~(2EMAJ%S˨;-Y˪nOǷZU ]h{}Z;":)m# R-WeNUٍ ffx,2jTS9I*$HW"pn)v~ SobZ;,H>tf/v~T3:d'SmU=peD;>Yg?+ДpNxIR>\ti_0-zyg|8-/<F^hu+):j#bWB|YMi f|wQeSk6}k{ aWycy>cM vXW]ƁnXp~ SܤM<(:r"-<p|+Ø&TCXɼ7I_Љ429ќEiΓL`6/$rNQLuٹ:Q-u'뺹{g @⻋+[}a6*uFoEu" t_# Pq'&P ]?ס prx:Rqs.F:~0F.FXtyX:QsOs +W3l^҅WER-8F 8}޳ x͘.# ع:Q}Rj&oRo'kxq}js/w9#ĄyOd]9We7j^W¯1(8F䏄 $Ip5v ]oob_j"UN`ԕE#zj÷|XLFSctӐ3)=;-l>yfKeOSe|,TC^NII>/}o=)zԓ9ް($@:I*N\?^Y N䧃]D$+{LZ\,IoN}OJ5s&{d=Qr%&3 y,px+1]𸕬;[?ns`V6/x,]RLnմ} TJvpAz<į|F6?XM95$j+gfRTdZWQ֦NoE|S%g@5+Aל\ӊ>7io'_J9OpFG *R*,&'WC7;y;갥e-;+쨯 (KwmzM<[r譮š~:VD߄VrMiC^LUahw%玝O#t=;F|\:6$kӗ}և U0J^ai!7o複CY=EJQLAO7p7|};NԏG"vlݐǞL•IۈmbWrlj1xku{+:i@@f -@fO^!=Iup)+.8WZt8TxS\d1AWy4鳆!KZyvmyB`/sZ7B^gsvhԧ_Ll*-y{Zv7]5EQrVXC \mtZ\ F5x$_w=ss1Z`jF:t +EZ!aHu^hc`DO`یV?GtuJS1œYy _M!e%oO! +endstream +endobj +270 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 269 0 R +/Resources 4 0 R +/Annots [ 271 0 R 272 0 R 273 0 R 274 0 R 275 0 R 276 0 R 277 0 R 278 0 R 279 0 R 280 0 R 281 0 R 282 0 R 283 0 R 284 0 R 285 0 R 286 0 R 287 0 R 288 0 R 289 0 R 290 0 R 291 0 R 292 0 R 293 0 R 294 0 R 295 0 R 296 0 R 297 0 R 298 0 R 299 0 R 300 0 R 301 0 R 302 0 R 303 0 R 304 0 R 305 0 R 306 0 R 307 0 R 308 0 R ] +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +271 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 799.173057 99.933071 780.423057 ] +/BS << +/W 0 +>> +/Dest (cb9-161) +>> +endobj +272 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 780.226036 99.933071 761.476036 ] +/BS << +/W 0 +>> +/Dest (cb9-162) +>> +endobj +273 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 761.279014 99.933071 742.529014 ] +/BS << +/W 0 +>> +/Dest (cb9-163) +>> +endobj +274 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 742.331993 99.933071 723.581993 ] +/BS << +/W 0 +>> +/Dest (cb9-164) +>> +endobj +275 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 723.384971 99.933071 704.634971 ] +/BS << +/W 0 +>> +/Dest (cb9-165) +>> +endobj +276 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 704.437950 99.933071 685.687950 ] +/BS << +/W 0 +>> +/Dest (cb9-166) +>> +endobj +277 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 685.490928 99.933071 666.740928 ] +/BS << +/W 0 +>> +/Dest (cb9-167) +>> +endobj +278 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 666.543907 99.933071 647.793907 ] +/BS << +/W 0 +>> +/Dest (cb9-168) +>> +endobj +279 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 647.596885 99.933071 628.846885 ] +/BS << +/W 0 +>> +/Dest (cb9-169) +>> +endobj +280 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 628.649864 99.933071 609.899864 ] +/BS << +/W 0 +>> +/Dest (cb9-170) +>> +endobj +281 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 609.702842 99.933071 590.952842 ] +/BS << +/W 0 +>> +/Dest (cb9-171) +>> +endobj +282 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 590.755821 99.933071 572.005821 ] +/BS << +/W 0 +>> +/Dest (cb9-172) +>> +endobj +283 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 571.808799 99.933071 553.058799 ] +/BS << +/W 0 +>> +/Dest (cb9-173) +>> +endobj +284 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 552.861778 99.933071 534.111778 ] +/BS << +/W 0 +>> +/Dest (cb9-174) +>> +endobj +285 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 533.914756 99.933071 515.164756 ] +/BS << +/W 0 +>> +/Dest (cb9-175) +>> +endobj +286 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 514.967735 99.933071 496.217735 ] +/BS << +/W 0 +>> +/Dest (cb9-176) +>> +endobj +287 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 496.020714 99.933071 477.270714 ] +/BS << +/W 0 +>> +/Dest (cb9-177) +>> +endobj +288 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 477.073692 99.933071 458.323692 ] +/BS << +/W 0 +>> +/Dest (cb9-178) +>> +endobj +289 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 458.126671 99.933071 439.376671 ] +/BS << +/W 0 +>> +/Dest (cb9-179) +>> +endobj +290 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 439.179649 99.933071 420.429649 ] +/BS << +/W 0 +>> +/Dest (cb9-180) +>> +endobj +291 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 420.232628 99.933071 401.482628 ] +/BS << +/W 0 +>> +/Dest (cb9-181) +>> +endobj +292 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 401.285606 99.933071 382.535606 ] +/BS << +/W 0 +>> +/Dest (cb9-182) +>> +endobj +293 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 382.338585 99.933071 363.588585 ] +/BS << +/W 0 +>> +/Dest (cb9-183) +>> +endobj +294 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 363.391563 99.933071 344.641563 ] +/BS << +/W 0 +>> +/Dest (cb9-184) +>> +endobj +295 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 344.444542 99.933071 325.694542 ] +/BS << +/W 0 +>> +/Dest (cb9-185) +>> +endobj +296 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 325.497520 99.933071 306.747520 ] +/BS << +/W 0 +>> +/Dest (cb9-186) +>> +endobj +297 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 306.550499 99.933071 287.800499 ] +/BS << +/W 0 +>> +/Dest (cb9-187) +>> +endobj +298 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 287.603477 99.933071 268.853477 ] +/BS << +/W 0 +>> +/Dest (cb9-188) +>> +endobj +299 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 268.656456 99.933071 249.906456 ] +/BS << +/W 0 +>> +/Dest (cb9-189) +>> +endobj +300 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 249.709434 99.933071 230.959434 ] +/BS << +/W 0 +>> +/Dest (cb9-190) +>> +endobj +301 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 230.762413 99.933071 212.012413 ] +/BS << +/W 0 +>> +/Dest (cb9-191) +>> +endobj +302 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 211.815391 99.933071 193.065391 ] +/BS << +/W 0 +>> +/Dest (cb9-192) +>> +endobj +303 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 192.868370 99.933071 174.118370 ] +/BS << +/W 0 +>> +/Dest (cb9-193) +>> +endobj +304 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 173.921348 99.933071 155.171348 ] +/BS << +/W 0 +>> +/Dest (cb9-194) +>> +endobj +305 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 154.974327 99.933071 136.224327 ] +/BS << +/W 0 +>> +/Dest (cb9-195) +>> +endobj +306 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 136.027305 99.933071 117.277305 ] +/BS << +/W 0 +>> +/Dest (cb9-196) +>> +endobj +307 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 117.080284 99.933071 98.330284 ] +/BS << +/W 0 +>> +/Dest (cb9-197) +>> +endobj +308 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 98.133262 99.933071 79.383262 ] +/BS << +/W 0 +>> +/Dest (cb9-198) +>> +endobj +309 0 obj +<< +/Filter /FlateDecode +/Length 4927 +>> +stream +x]߫帑~?y"<,@``B=73 ˝d߯t%٪o{%T>}e5~3'9R1/_oZEX]ow]TPT1{6}[s'9R1Ũ IޫBR0 +3|t֍If3IHJ0\44&eaqR0G!hab ΑҌW4kTHf1-"D+FJ*`Hix㜖`[(`M[,Z' i ]IL-:h#eR>'Ĵ4bӬ'"- CJiAbZBb#qb)0%.+v$.H3Nₘa T̥4[dK\OiH\,l7(qA Qf`F,Ҝ83I\\P*J\aHL)Є~l"` y'qAJ$qAJj+qf kO€b7Y8Ҵ x: 0.9Ld҈7 6Sx3J\lpQ4) x$b fnMx@x3]LRxi7JLNLq&q@z=Vq@{1[Eq@|9F `&"ULH4Z\6_cnSD93EɌaטed{o{~]Y)[~ O.p`y،]RCGUX?F܌4m_dX683\_/" zX,[i +0{|B]0;Sa)*b_ e܉WUt0zVXݘ_nOO?>3oW7zȽ̃4orᙸϷ?~Z"n bf[:‹SP(kg[8)M_䇎[䙆ɞ׾J>\KpKt&#T.K)T7am؆~2GsVbhÄ߲s:iZ+֑s#`hj5ihLsYd?D9_N|ûwƜu:#8> *-iˎ-OMH:PT0vȠH`frrP^9ˀrTj\ +bۂlnE3W#4W Ž bac~]9 4^#"m'܁%3yZud^gR:A 1m B pp_\(BՃd"^]!<ÈD%n)t?Eg@:!4[ȸuΣݏ&;^X/hόn= z B &^,PnWؙӹx"1B3*N/@cbzp(u ީ= IPWVWg=Iw7$1{u<Ǵ *Χm2%MO)cռQ{XowNQ)H&ٲǽb&u,Y8YLgkU~#e3eN_9ҵW@J3z{s0Lef``HI[Z~|9";||& W~;yf#¬ϼޛ_Bx} ibϕgzD[;v(YAɔy0GJ +7sHGޤ]8YW@2k0ɍǍS~`@@ +^(΃jN2j]篁fY^pݵ +Z@i>%AJXȸߤu1dgBiUiF x~1O߃!^ՠ !0+\z΄o{tF8.t`'tA寽8?qFОh7&IpO:ׁ'o Lx + ++7uhNhq8ws:p,~1 uԿFpݨ}hzQ-o9%z rR:Wc%\/2׵g]m}ƺ_s뜸 nA`iCz~uYNUCe5: 3@kOE=[罶IOЙ_>hB 买Xmk:ⵍǼ6AP篓:Tv]9<@ -lͳhő''Ԃ WBmR:P#~9s&w ]=U mh 6hڨ4oǀ8a59@[zhy`cG|Q&+fms_ԹO>sҺkzY6ղ[m>1j[ *Miޛӹ 0~g޺3vf#7jc>A`p>0"AIṀBlX,lcҊ_mxZĎ3kb=9m6syvbgn9Z^Na~K1P!f2ۨ[D':XӚe<`Bgv_$g>rɆWO\VF|yZzV}m m}"cԞ Z!2Gu7s@85 UӀ<C[y͂ {s|i6pA7syvݜh]A BCn\%6Sޚ &uܪвˆDpaH0j =Yrfl0seo9P.t`.5 FcႠl2!1|2=Owa5sX b8sѺ+h{(Kn.D'Bof3['>Z:`hF홀J& +Z߃C6gw`Ѥu yꇹ2ݪo`euh{XǑFFS">gtI)[d|@ZzH WN +8sGێ5j{K`fgeӟdO{7r1Y,$ g2'XhX3o߷DZ%O9Q<+o+)sDބ~ꮺ&rkCXMtݿ䵱= V.J;IKQK\d/9kO&KkaZjDo71߳4_{*\) j FOiO0vcDZ֩Ío7~d|r/ԕ[I74iLKSah.2>}PRfqRv*YF塲Zw~Ac$-gҭk]u*yŦ–\ bkb_7;}^ʂkM*wX{SLln+f觥& :a3 čFߋxkdҌvLse?ׄڙHN +#ѥnIǸq5;#eޑ_'tE-Y=Xᖎ5hN+8c\ȩCEv6/PğڈK:e%O +endstream +endobj +310 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 309 0 R +/Resources 4 0 R +/Annots [ 311 0 R 312 0 R 313 0 R 314 0 R 315 0 R 316 0 R 317 0 R 318 0 R 319 0 R 320 0 R 321 0 R 322 0 R 323 0 R 324 0 R 325 0 R 326 0 R 327 0 R 328 0 R 329 0 R 330 0 R 331 0 R 332 0 R 333 0 R 334 0 R 335 0 R 336 0 R 337 0 R 338 0 R 339 0 R 340 0 R 341 0 R 342 0 R 343 0 R 344 0 R 345 0 R 346 0 R 347 0 R 348 0 R ] +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +311 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 799.173057 99.933071 780.423057 ] +/BS << +/W 0 +>> +/Dest (cb9-199) +>> +endobj +312 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 780.226036 99.933071 761.476036 ] +/BS << +/W 0 +>> +/Dest (cb9-200) +>> +endobj +313 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 761.279014 99.933071 742.529014 ] +/BS << +/W 0 +>> +/Dest (cb9-201) +>> +endobj +314 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 742.331993 99.933071 723.581993 ] +/BS << +/W 0 +>> +/Dest (cb9-202) +>> +endobj +315 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 723.384971 99.933071 704.634971 ] +/BS << +/W 0 +>> +/Dest (cb9-203) +>> +endobj +316 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 704.437950 99.933071 685.687950 ] +/BS << +/W 0 +>> +/Dest (cb9-204) +>> +endobj +317 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 685.490928 99.933071 666.740928 ] +/BS << +/W 0 +>> +/Dest (cb9-205) +>> +endobj +318 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 666.543907 99.933071 647.793907 ] +/BS << +/W 0 +>> +/Dest (cb9-206) +>> +endobj +319 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 647.596885 99.933071 628.846885 ] +/BS << +/W 0 +>> +/Dest (cb9-207) +>> +endobj +320 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 628.649864 99.933071 609.899864 ] +/BS << +/W 0 +>> +/Dest (cb9-208) +>> +endobj +321 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 609.702842 99.933071 590.952842 ] +/BS << +/W 0 +>> +/Dest (cb9-209) +>> +endobj +322 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 590.755821 99.933071 572.005821 ] +/BS << +/W 0 +>> +/Dest (cb9-210) +>> +endobj +323 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 571.808799 99.933071 553.058799 ] +/BS << +/W 0 +>> +/Dest (cb9-211) +>> +endobj +324 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 552.861778 99.933071 534.111778 ] +/BS << +/W 0 +>> +/Dest (cb9-212) +>> +endobj +325 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 533.914756 99.933071 515.164756 ] +/BS << +/W 0 +>> +/Dest (cb9-213) +>> +endobj +326 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 514.967735 99.933071 496.217735 ] +/BS << +/W 0 +>> +/Dest (cb9-214) +>> +endobj +327 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 496.020714 99.933071 477.270714 ] +/BS << +/W 0 +>> +/Dest (cb9-215) +>> +endobj +328 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 477.073692 99.933071 458.323692 ] +/BS << +/W 0 +>> +/Dest (cb9-216) +>> +endobj +329 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 458.126671 99.933071 439.376671 ] +/BS << +/W 0 +>> +/Dest (cb9-217) +>> +endobj +330 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 439.179649 99.933071 420.429649 ] +/BS << +/W 0 +>> +/Dest (cb9-218) +>> +endobj +331 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 420.232628 99.933071 401.482628 ] +/BS << +/W 0 +>> +/Dest (cb9-219) +>> +endobj +332 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 401.285606 99.933071 382.535606 ] +/BS << +/W 0 +>> +/Dest (cb9-220) +>> +endobj +333 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 382.338585 99.933071 363.588585 ] +/BS << +/W 0 +>> +/Dest (cb9-221) +>> +endobj +334 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 363.391563 99.933071 344.641563 ] +/BS << +/W 0 +>> +/Dest (cb9-222) +>> +endobj +335 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 344.444542 99.933071 325.694542 ] +/BS << +/W 0 +>> +/Dest (cb9-223) +>> +endobj +336 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 325.497520 99.933071 306.747520 ] +/BS << +/W 0 +>> +/Dest (cb9-224) +>> +endobj +337 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 306.550499 99.933071 287.800499 ] +/BS << +/W 0 +>> +/Dest (cb9-225) +>> +endobj +338 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 287.603477 99.933071 268.853477 ] +/BS << +/W 0 +>> +/Dest (cb9-226) +>> +endobj +339 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 268.656456 99.933071 249.906456 ] +/BS << +/W 0 +>> +/Dest (cb9-227) +>> +endobj +340 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 249.709434 99.933071 230.959434 ] +/BS << +/W 0 +>> +/Dest (cb9-228) +>> +endobj +341 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 230.762413 99.933071 212.012413 ] +/BS << +/W 0 +>> +/Dest (cb9-229) +>> +endobj +342 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 211.815391 99.933071 193.065391 ] +/BS << +/W 0 +>> +/Dest (cb9-230) +>> +endobj +343 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 192.868370 99.933071 174.118370 ] +/BS << +/W 0 +>> +/Dest (cb9-231) +>> +endobj +344 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 173.921348 99.933071 155.171348 ] +/BS << +/W 0 +>> +/Dest (cb9-232) +>> +endobj +345 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 154.974327 99.933071 136.224327 ] +/BS << +/W 0 +>> +/Dest (cb9-233) +>> +endobj +346 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 136.027305 99.933071 117.277305 ] +/BS << +/W 0 +>> +/Dest (cb9-234) +>> +endobj +347 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 117.080284 99.933071 98.330284 ] +/BS << +/W 0 +>> +/Dest (cb9-235) +>> +endobj +348 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 98.133262 99.933071 79.383262 ] +/BS << +/W 0 +>> +/Dest (cb9-236) +>> +endobj +349 0 obj +<< +/Filter /FlateDecode +/Length 4954 +>> +stream +x]]6}_ρ(J >$,w}ؿܖ,[nΐѩ*IU%gTJ9w}rUM%VSO?.*orH(z_˶_]1Wr8ˑ)F텬,mH^h}QŜ,ᣳϊP,MBV䢡>+C > +YFCcvg + YB6yy&Z)H7R^V!<]|F純](B]l$Eb;YKJbtA)/99'eSgW>i)M00H! +y] 2&spYGV!lm\ƧMC4Rm *Pw#PAAvNy/݁(K6 +#ɜDw|V ACW^6N8ˬj O)aW&9X^f@9>]W:1N@h:wNeiAzdWuY1 ;YR ʜ8 kx"pVoڙj 'z%vyl-2k8(sangoU}8$3gy:KkwE|d9h>Z6 1Øg|&Voke '&>/!umX'}`.:paP<3|>!}&V1v3fC *h@{b20coֺ`M@ڰW6ւKz=yԑ'L"mP<`㧗"_`sc&JM^&J/ Jؐ>.he{P<ʀJ߷&J.Ke?u 'uz=femaeu]Zs|o,g_9$:O҉'[~ <7D=9շfKl.2ZiL4p +^<8+݃ӽ(uV 06Ah. +Q$c2/LX*mMB}l=.;s;p(N#p{cpExA)ʛaCFr958'pj652ĦwmhYz=oS+v+lZ==&rְ^˳*]jE9K;-P0\<{P">k,,Z뾽u_A~&򫼮nB4!Ldħ"5 3RժoY4q?7oԞҍg]6a\)}t>T&~7؜^)K6} /Cd1(kɄ5&ƇhP}VD<8]IN/my 1@'k~c Jo Ӽ_ru4Æoꥏ.A~Lde]AL0*= oR748K `P8"0(syPtf^4<][Gꥏ=%mWD. ʜPr?PJ7sW>G{cˁ2S `CiC"SV'.r͋,,w{6:lVn~ߺJ_ a㮗axXAO:$;@{|Olnu: ŷۏ_z~l3IqVmL^3uޑ>6[-#I')xvb.40_.u=2oߊS2nzo9!Lu)M Ty[\+'_~Z_}w8rX5,$B2>kN7%EU5qеe +ɼw$>Q7-FR֟9mיfKYK|YڦRU]XIF\S})Yq^.B׺TqoFߕKLH˥r͵:RJh.=@Rx!m959\'2u.§׋Bipzm"5,1,iqF2L?,WK#<7kPYI=5]*ꎱb/PI=JmI$Ucl*j2SXʕm.J%W(=}^ƂikKr&:?P6//asd~*,ɼtXSk@ +KBSrsd}z̜%oE dKFwdC/cyu͎[]Vn]%{;x%N&Ѻdqkv^4AMm҇Mе;/^`.х.eZh1צJ\qŤ#Fml̋f+5m#Yq[EF%-yn1m%m\`j/ڊi+}<*28R{M}mSqj%<9˰62yuQ3:9X\ h1ڤEl.F]I[5Z E VkE0n8OoGvoG%tebD-rwz)Y}$V$mnXksnQ$,^|WMs٬Hʟiäc30D4}?X!?]j4q+em á-8|%ZǸ[:Bs[>慜z9T/;?XtZ>yP۷ +endstream +endobj +350 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 349 0 R +/Resources 4 0 R +/Annots [ 351 0 R 352 0 R 353 0 R 354 0 R 355 0 R 356 0 R 357 0 R 358 0 R 359 0 R 360 0 R 361 0 R 362 0 R 363 0 R 364 0 R 365 0 R 366 0 R 367 0 R 368 0 R 369 0 R 370 0 R 371 0 R 372 0 R 373 0 R 374 0 R 375 0 R 376 0 R 377 0 R 378 0 R 379 0 R 380 0 R 381 0 R 382 0 R 383 0 R 384 0 R 385 0 R 386 0 R 387 0 R 388 0 R ] +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +351 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 799.173057 99.933071 780.423057 ] +/BS << +/W 0 +>> +/Dest (cb9-237) +>> +endobj +352 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 780.226036 99.933071 761.476036 ] +/BS << +/W 0 +>> +/Dest (cb9-238) +>> +endobj +353 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 761.279014 99.933071 742.529014 ] +/BS << +/W 0 +>> +/Dest (cb9-239) +>> +endobj +354 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 742.331993 99.933071 723.581993 ] +/BS << +/W 0 +>> +/Dest (cb9-240) +>> +endobj +355 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 723.384971 99.933071 704.634971 ] +/BS << +/W 0 +>> +/Dest (cb9-241) +>> +endobj +356 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 704.437950 99.933071 685.687950 ] +/BS << +/W 0 +>> +/Dest (cb9-242) +>> +endobj +357 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 685.490928 99.933071 666.740928 ] +/BS << +/W 0 +>> +/Dest (cb9-243) +>> +endobj +358 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 666.543907 99.933071 647.793907 ] +/BS << +/W 0 +>> +/Dest (cb9-244) +>> +endobj +359 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 647.596885 99.933071 628.846885 ] +/BS << +/W 0 +>> +/Dest (cb9-245) +>> +endobj +360 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 628.649864 99.933071 609.899864 ] +/BS << +/W 0 +>> +/Dest (cb9-246) +>> +endobj +361 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 609.702842 99.933071 590.952842 ] +/BS << +/W 0 +>> +/Dest (cb9-247) +>> +endobj +362 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 590.755821 99.933071 572.005821 ] +/BS << +/W 0 +>> +/Dest (cb9-248) +>> +endobj +363 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 571.808799 99.933071 553.058799 ] +/BS << +/W 0 +>> +/Dest (cb9-249) +>> +endobj +364 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 552.861778 99.933071 534.111778 ] +/BS << +/W 0 +>> +/Dest (cb9-250) +>> +endobj +365 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 533.914756 99.933071 515.164756 ] +/BS << +/W 0 +>> +/Dest (cb9-251) +>> +endobj +366 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 514.967735 99.933071 496.217735 ] +/BS << +/W 0 +>> +/Dest (cb9-252) +>> +endobj +367 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 496.020714 99.933071 477.270714 ] +/BS << +/W 0 +>> +/Dest (cb9-253) +>> +endobj +368 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 477.073692 99.933071 458.323692 ] +/BS << +/W 0 +>> +/Dest (cb9-254) +>> +endobj +369 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 458.126671 99.933071 439.376671 ] +/BS << +/W 0 +>> +/Dest (cb9-255) +>> +endobj +370 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 439.179649 99.933071 420.429649 ] +/BS << +/W 0 +>> +/Dest (cb9-256) +>> +endobj +371 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 420.232628 99.933071 401.482628 ] +/BS << +/W 0 +>> +/Dest (cb9-257) +>> +endobj +372 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 401.285606 99.933071 382.535606 ] +/BS << +/W 0 +>> +/Dest (cb9-258) +>> +endobj +373 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 382.338585 99.933071 363.588585 ] +/BS << +/W 0 +>> +/Dest (cb9-259) +>> +endobj +374 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 363.391563 99.933071 344.641563 ] +/BS << +/W 0 +>> +/Dest (cb9-260) +>> +endobj +375 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 344.444542 99.933071 325.694542 ] +/BS << +/W 0 +>> +/Dest (cb9-261) +>> +endobj +376 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 325.497520 99.933071 306.747520 ] +/BS << +/W 0 +>> +/Dest (cb9-262) +>> +endobj +377 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 306.550499 99.933071 287.800499 ] +/BS << +/W 0 +>> +/Dest (cb9-263) +>> +endobj +378 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 287.603477 99.933071 268.853477 ] +/BS << +/W 0 +>> +/Dest (cb9-264) +>> +endobj +379 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 268.656456 99.933071 249.906456 ] +/BS << +/W 0 +>> +/Dest (cb9-265) +>> +endobj +380 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 249.709434 99.933071 230.959434 ] +/BS << +/W 0 +>> +/Dest (cb9-266) +>> +endobj +381 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 230.762413 99.933071 212.012413 ] +/BS << +/W 0 +>> +/Dest (cb9-267) +>> +endobj +382 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 211.815391 99.933071 193.065391 ] +/BS << +/W 0 +>> +/Dest (cb9-268) +>> +endobj +383 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 192.868370 99.933071 174.118370 ] +/BS << +/W 0 +>> +/Dest (cb9-269) +>> +endobj +384 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 173.921348 99.933071 155.171348 ] +/BS << +/W 0 +>> +/Dest (cb9-270) +>> +endobj +385 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 154.974327 99.933071 136.224327 ] +/BS << +/W 0 +>> +/Dest (cb9-271) +>> +endobj +386 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 136.027305 99.933071 117.277305 ] +/BS << +/W 0 +>> +/Dest (cb9-272) +>> +endobj +387 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 117.080284 99.933071 98.330284 ] +/BS << +/W 0 +>> +/Dest (cb9-273) +>> +endobj +388 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 98.133262 99.933071 79.383262 ] +/BS << +/W 0 +>> +/Dest (cb9-274) +>> +endobj +389 0 obj +<< +/Filter /FlateDecode +/Length 5112 +>> +stream +x]]$9r}_Q `20{a]ۆY?GfFU05wʐB"$e=|gN9E:_~>)9OMl )9)RJ4.[\5 iʸ'!;iiL%d f ބG. ěI]. fL@ V 7$gz"Kꊁ̭ꒁbJꚁJ.LDULHF]60_cSD9dư x5hl&_wJY(u!2t5HJB1"f-.nb$bL\?صɰXlpf;^D +]n:Oo#do$ +|aǔ~mfR}\YR1 y08^1VeJ1>LuE7]yJՊJ_gnyYEReun10\ttP qhr`S~ e 6a}"Ve*,\:]=i,z Bp<-~1uf ֳbPu^G9v'.) r,DnAQ/^ + +Lpb1<{5>Fx~.d͝Lx1ׁ@ +<.tnfnDVG)00 7Avywv DF6eJ$@g3ÚxArLu dS{&$WT`ò41 Mzqڏh]>1P=[ Y"Ry9YB9[攷9`&7٤Tc⩱'i|N8r~_}=8{<>Ve||9fgduԹy޷l3a*Z6| ;/m16Wzo#^1WZѿ\MBt!ׁZbdB8D6uC]m#OweVܫpd'cޥ$>,G@@Nԫ*zkqͽPY1Pk#!G'NΎY2hoPuaBc9ĺй}ee .xpc5FyS|//p8fC$M{,͚){ϫM9mePud V~&Zy8Xdَi]Mȕz mǰl+{푉Y_k{^,ojewP-(XBWP#r][;^ Z}^fɓ6*|R_.%)Q~VrtKUSp`GPZ|s luK;\]0ݱE/vtP\mKhp7cVGɏNM;c0.]sg'-Y{m.ΈrW*Zyk;k?/ݫEoI{ܶݴ>}V[mמsk5鿥b.r헥ftޯnwi7.}b +v++uWu%9`/ſcYv%5[7M;70~i_)Wol/Qz{v~ӛ?ʣzmZ[ g?pqm[{6۟!_ ?.76cpܥX`pP@+ɖEo0>0Ѿ= oG2=fs{c[:mCh=S;9nPu(S&K[~K SI~!PvX>c'k2"ZAPR= rO2vf~0>(t^q~S%\ӽQ6`2"8=#y< 9u_!I,Ks:;ِD~]å?] '㰁(-;ȗ!o+=7n񜔳ߺЇYl`߮mz^{ǘq)29(t G?۝\>vjk\MU_An,1<: Xy73 gI + thڝ9 Ns<^YHbg7)WpяAd66?u }j\axgoCdxbo-HRv~t>""2v$t5VWu"EZV> !}eA+e2kga mjejPFVsQuXgyTiy=Ńڡ)YZ-$Y,ʑS/0=t=f!!Ϧ=q]R6ϧ8/;6ҨN ጀIΜ|k y |Csk. 7MK87vr27]h|)bv@,壩eJSa2vY\]?M?5ӕ+bEhb{3E,__b54H}V`S<bt5~̵!Mg/y]rB~+}WRTi]gwF* %g.'E)E:w& +j\)IͷjOj.-LiP\^]I#Ɨkq(}mÍ /@u$6)$%wh\aB^/]U̦IeK CJJy*%U"c׍ceZ@f*f¼ҭk]uyBR8qwdD?':Xވ{ǝl*L&_gÃbz"æ:h-u|MM4b/9meP[S{ahm؉]kpGxٴbud~iߴ zI( R.Vᢿ\rKmJ>wEJ):E\ⲫ07Vf + JjӋw!Qfl. \`_++{?<~q/&2DZJ{i1Vӥ~:9,P-wcW' Zh'妶d}TdJDQ2SdTlZiYmT}&[ҡڈ+zK:@sjwgZK"vRl_7. j晚v9?;TZ3`XnD-u[qw8Fp7$nIakׄ{|n;qBWTG]M[z%FsoBNL*[=DXuIg^jɫ]@_ +endstream +endobj +390 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 389 0 R +/Resources 4 0 R +/Annots [ 391 0 R 392 0 R 393 0 R 394 0 R 395 0 R 396 0 R 397 0 R 398 0 R 399 0 R 400 0 R 401 0 R 402 0 R 403 0 R 404 0 R 405 0 R 406 0 R 407 0 R 408 0 R 409 0 R 410 0 R 411 0 R 412 0 R 413 0 R 414 0 R 415 0 R 416 0 R 417 0 R 418 0 R 419 0 R 420 0 R 421 0 R 422 0 R 423 0 R ] +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +391 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 799.173057 99.933071 780.423057 ] +/BS << +/W 0 +>> +/Dest (cb9-275) +>> +endobj +392 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 780.226036 99.933071 761.476036 ] +/BS << +/W 0 +>> +/Dest (cb9-276) +>> +endobj +393 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 761.279014 99.933071 742.529014 ] +/BS << +/W 0 +>> +/Dest (cb9-277) +>> +endobj +394 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 742.331993 99.933071 723.581993 ] +/BS << +/W 0 +>> +/Dest (cb9-278) +>> +endobj +395 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 723.384971 99.933071 704.634971 ] +/BS << +/W 0 +>> +/Dest (cb9-279) +>> +endobj +396 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 704.437950 99.933071 685.687950 ] +/BS << +/W 0 +>> +/Dest (cb9-280) +>> +endobj +397 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 685.490928 99.933071 666.740928 ] +/BS << +/W 0 +>> +/Dest (cb9-281) +>> +endobj +398 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 666.543907 99.933071 647.793907 ] +/BS << +/W 0 +>> +/Dest (cb9-282) +>> +endobj +399 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 647.596885 99.933071 628.846885 ] +/BS << +/W 0 +>> +/Dest (cb9-283) +>> +endobj +400 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 628.649864 99.933071 609.899864 ] +/BS << +/W 0 +>> +/Dest (cb9-284) +>> +endobj +401 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 609.702842 99.933071 590.952842 ] +/BS << +/W 0 +>> +/Dest (cb9-285) +>> +endobj +402 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 590.755821 99.933071 572.005821 ] +/BS << +/W 0 +>> +/Dest (cb9-286) +>> +endobj +403 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 571.808799 99.933071 553.058799 ] +/BS << +/W 0 +>> +/Dest (cb9-287) +>> +endobj +404 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 552.861778 99.933071 534.111778 ] +/BS << +/W 0 +>> +/Dest (cb9-288) +>> +endobj +405 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 84.259843 443.811778 177.650223 428.811778 ] +/BS << +/W 0 +>> +/A << +/Type /Action +/S /URI +/URI (file:///doc_tools/%7B%7B%3Cbaseurl%3E%7D%7D/rancher/v2.6/en/installation) +>> +>> +endobj +406 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 415.114756 101.433071 396.364756 ] +/BS << +/W 0 +>> +/Dest (cb10-1) +>> +endobj +407 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 396.167735 99.933071 377.417735 ] +/BS << +/W 0 +>> +/Dest (cb10-2) +>> +endobj +408 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 377.220714 99.933071 358.470714 ] +/BS << +/W 0 +>> +/Dest (cb10-3) +>> +endobj +409 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 358.273692 99.933071 339.523692 ] +/BS << +/W 0 +>> +/Dest (cb10-4) +>> +endobj +410 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 339.326671 99.933071 320.576671 ] +/BS << +/W 0 +>> +/Dest (cb10-5) +>> +endobj +411 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 320.379649 99.933071 301.629649 ] +/BS << +/W 0 +>> +/Dest (cb10-6) +>> +endobj +412 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 301.432628 99.933071 282.682628 ] +/BS << +/W 0 +>> +/Dest (cb10-7) +>> +endobj +413 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 282.485606 99.933071 263.735606 ] +/BS << +/W 0 +>> +/Dest (cb10-8) +>> +endobj +414 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 263.538585 99.933071 244.788585 ] +/BS << +/W 0 +>> +/Dest (cb10-9) +>> +endobj +415 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 244.591563 99.933071 225.841563 ] +/BS << +/W 0 +>> +/Dest (cb10-10) +>> +endobj +416 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 225.644542 99.933071 206.894542 ] +/BS << +/W 0 +>> +/Dest (cb10-11) +>> +endobj +417 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 206.697520 99.933071 187.947520 ] +/BS << +/W 0 +>> +/Dest (cb10-12) +>> +endobj +418 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 187.750499 99.933071 169.000499 ] +/BS << +/W 0 +>> +/Dest (cb10-13) +>> +endobj +419 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 168.803477 99.933071 150.053477 ] +/BS << +/W 0 +>> +/Dest (cb10-14) +>> +endobj +420 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 149.856456 99.933071 131.106456 ] +/BS << +/W 0 +>> +/Dest (cb10-15) +>> +endobj +421 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 130.909434 99.933071 112.159434 ] +/BS << +/W 0 +>> +/Dest (cb10-16) +>> +endobj +422 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 111.962413 99.933071 93.212413 ] +/BS << +/W 0 +>> +/Dest (cb10-17) +>> +endobj +423 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 93.015391 99.933071 74.265391 ] +/BS << +/W 0 +>> +/Dest (cb10-18) +>> +endobj +424 0 obj +<< +/Filter /FlateDecode +/Length 4783 +>> +stream +x]ߋ8~Ͽ"F+)B`eaazjw9jfiɲ.+wQ]V>²-~~p/395)XJ.&_YN&s*J&S/zqwoJޅ?I9%V:RX!| ++:[,EcA󞕠4`P89ڳ +ɱƂ[e&\05w&K5]pcK(i]c0"tHVAi]`&9*P` r ]7e4,( Ӱh$Lְ0aƥ7,֛X)PfҰ0t QTtATq Lz"t5,(1z3kX0MDjJ z3n(@&Ed5,֛lLgfYqf5c`s2^VIXԤ2G5kiLgմ"&JWba 6iG%4}N#D; D91Z^O7 G~]vjc.gtO~OB>Ap?:`IaUR G5yu2!\-n_돓?M5;#\CGH:2YD/jc"2V[I!޳.Ͻo& ?y4"i#KR׉&L'F2 7ƗQ`˄jt/,EE BXvLجhQJΗu$io/b9ziֽQfDW!lS"C]6_2;I,ed.|)ESGif7[ͷ8N6]iwT_Ni&=pI?(&G/j7$vA븠DB4,Q\قkm>upCH<ϡ\$IXo}\s`P;O4(Ss\leǑ6wEG,I6OgP:U»ͳ;33n ݯR]I\0[:-`Ȼ-tgi1tla +󁇾+3y9yA|Q8y4i9aS+\ƌ=>>u#66tӴ.7]sݳŽ<[K?lQwălF^d0*䳍72qYy"#[o֍|>PB.[sLfEzĥc_JE.WncvzJu\95}W]5yN4cԿ"5:ml"h#]afAk0 "C N`NR;͹ \ǽ]v[^FrfE\!iX{َnN'oeSj kള˝LfzW?ve8/)6y)!*<ۘ3(s B鼫J'=_^hzw-e$K0r|;1tM))hmhp[Ox=ul]Pn9ڶQ/>ʦ69hxDH\Y6Y9!aJ>h4;(sӹ:$~n]`ET%)"Gz;Ng\II_}~ݶXVOGE7YnE4(snD)Û_R3yu>n{V~f%wdx,Y NC1f?wH`j/yƟP䓎Nא)g>xqŞj|!m-@:d96ȉE.TO͹Sql7r?vu3?}. kl\G9@h*s e.1v㶻 F7k4,Ǧ%,7sC@rzZ)$+ Kxb.d_oNQ|q:Ȳܒ_or:=+dL#~,mu ؐLٖ`)ԏ>ԻP@:rC?n|ǽmhL&ߴ+؆MGBvy)7yy ג!kܞ>ȾwLx|m qIٲ[E; )U+)LqWˮhRSouϽ)H.0B.i2UWt]_-+}=ʱ6hoמnL t*Mq}.PG-YZю@J[5Z'Jk5SjLw^J-X{:Viz]Yx-Zޝrwv)bɬb/ݨJ۷l{nEE,J 3v2lVL 1QKi:c|Hܥ4y;DZdGa&'= +F9gtU%drKuY`28ӗOߜiI]s$AoI֩))e>~A +endstream +endobj +425 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 424 0 R +/Resources 4 0 R +/Annots [ 426 0 R 427 0 R 428 0 R 429 0 R 430 0 R 431 0 R 432 0 R 433 0 R 434 0 R 435 0 R 436 0 R 437 0 R 438 0 R 439 0 R 440 0 R 441 0 R 442 0 R 443 0 R 444 0 R 445 0 R 446 0 R 447 0 R 448 0 R 449 0 R 450 0 R 451 0 R 452 0 R 453 0 R 454 0 R 455 0 R 456 0 R 457 0 R 458 0 R 459 0 R 460 0 R 461 0 R 462 0 R ] +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +426 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 799.173057 99.933071 780.423057 ] +/BS << +/W 0 +>> +/Dest (cb10-19) +>> +endobj +427 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 780.226036 99.933071 761.476036 ] +/BS << +/W 0 +>> +/Dest (cb10-20) +>> +endobj +428 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 761.279014 99.933071 742.529014 ] +/BS << +/W 0 +>> +/Dest (cb10-21) +>> +endobj +429 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 742.331993 99.933071 723.581993 ] +/BS << +/W 0 +>> +/Dest (cb10-22) +>> +endobj +430 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 723.384971 99.933071 704.634971 ] +/BS << +/W 0 +>> +/Dest (cb10-23) +>> +endobj +431 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 704.437950 99.933071 685.687950 ] +/BS << +/W 0 +>> +/Dest (cb10-24) +>> +endobj +432 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 685.490928 99.933071 666.740928 ] +/BS << +/W 0 +>> +/Dest (cb10-25) +>> +endobj +433 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 666.543907 99.933071 647.793907 ] +/BS << +/W 0 +>> +/Dest (cb10-26) +>> +endobj +434 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 647.596885 99.933071 628.846885 ] +/BS << +/W 0 +>> +/Dest (cb10-27) +>> +endobj +435 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 628.649864 99.933071 609.899864 ] +/BS << +/W 0 +>> +/Dest (cb10-28) +>> +endobj +436 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 609.702842 99.933071 590.952842 ] +/BS << +/W 0 +>> +/Dest (cb10-29) +>> +endobj +437 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 571.808799 99.933071 553.058799 ] +/BS << +/W 0 +>> +/Dest (cb10-30) +>> +endobj +438 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 552.861778 99.933071 534.111778 ] +/BS << +/W 0 +>> +/Dest (cb10-31) +>> +endobj +439 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 533.914756 99.933071 515.164756 ] +/BS << +/W 0 +>> +/Dest (cb10-32) +>> +endobj +440 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 514.967735 99.933071 496.217735 ] +/BS << +/W 0 +>> +/Dest (cb10-33) +>> +endobj +441 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 496.020714 99.933071 477.270714 ] +/BS << +/W 0 +>> +/Dest (cb10-34) +>> +endobj +442 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 477.073692 99.933071 458.323692 ] +/BS << +/W 0 +>> +/Dest (cb10-35) +>> +endobj +443 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 458.126671 99.933071 439.376671 ] +/BS << +/W 0 +>> +/Dest (cb10-36) +>> +endobj +444 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 439.179649 99.933071 420.429649 ] +/BS << +/W 0 +>> +/Dest (cb10-37) +>> +endobj +445 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 420.232628 99.933071 401.482628 ] +/BS << +/W 0 +>> +/Dest (cb10-38) +>> +endobj +446 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 401.285606 99.933071 382.535606 ] +/BS << +/W 0 +>> +/Dest (cb10-39) +>> +endobj +447 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 382.338585 99.933071 363.588585 ] +/BS << +/W 0 +>> +/Dest (cb10-40) +>> +endobj +448 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 363.391563 99.933071 344.641563 ] +/BS << +/W 0 +>> +/Dest (cb10-41) +>> +endobj +449 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 344.444542 99.933071 325.694542 ] +/BS << +/W 0 +>> +/Dest (cb10-42) +>> +endobj +450 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 325.497520 99.933071 306.747520 ] +/BS << +/W 0 +>> +/Dest (cb10-43) +>> +endobj +451 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 306.550499 99.933071 287.800499 ] +/BS << +/W 0 +>> +/Dest (cb10-44) +>> +endobj +452 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 287.603477 99.933071 268.853477 ] +/BS << +/W 0 +>> +/Dest (cb10-45) +>> +endobj +453 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 268.656456 99.933071 249.906456 ] +/BS << +/W 0 +>> +/Dest (cb10-46) +>> +endobj +454 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 249.709434 99.933071 230.959434 ] +/BS << +/W 0 +>> +/Dest (cb10-47) +>> +endobj +455 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 230.762413 99.933071 212.012413 ] +/BS << +/W 0 +>> +/Dest (cb10-48) +>> +endobj +456 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 211.815391 99.933071 193.065391 ] +/BS << +/W 0 +>> +/Dest (cb10-49) +>> +endobj +457 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 192.868370 99.933071 174.118370 ] +/BS << +/W 0 +>> +/Dest (cb10-50) +>> +endobj +458 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 173.921348 99.933071 155.171348 ] +/BS << +/W 0 +>> +/Dest (cb10-51) +>> +endobj +459 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 154.974327 99.933071 136.224327 ] +/BS << +/W 0 +>> +/Dest (cb10-52) +>> +endobj +460 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 136.027305 99.933071 117.277305 ] +/BS << +/W 0 +>> +/Dest (cb10-53) +>> +endobj +461 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 117.080284 99.933071 98.330284 ] +/BS << +/W 0 +>> +/Dest (cb10-54) +>> +endobj +462 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 98.133262 99.933071 79.383262 ] +/BS << +/W 0 +>> +/Dest (cb10-55) +>> +endobj +463 0 obj +<< +/Filter /FlateDecode +/Length 4628 +>> +stream +x]ߋ$ ~￢g[/8,I`N6!.yȿOeZ=Cv..,dD# FH 4`89ec1\ Ia9 9VAL^`":=j'x xNQxʃ,6Z +P2+ٜ-`B&9`0,ц K 5AƏNJ9+&KN}bY| |,6E 0>%ų>YCH_Xdxi@@, ְ4 z G `(ch  z8Y߃Ix%+pʘZK԰ w]T^1.Ǖq^6.ƢaAQ  <װ wSk#ˌVxhjV̬aAh$T԰&RAЏ$A$?H6 x;`'GQuo\O' ~rޔ񷼐Ϗ?ѳsWƇc P79k!gzBce,; +. =#{E@qztF{ڕ蔟g]zl) Už*jFU EvdsA.㙼;6+-(&Zn|>z#UѲmGƋ2M1=L U֏|!9Rnq+"۸R$oŘJ~ckah{{=o,KUHϓD RN;;]>X8]LŒ%G9auʯLH`<,3V8u ˏvpe9Z 9ĘajZIfgdm18jd;Bވӻ 2 w;=$Xttf;u$mnz[9#/J[ (UaeU92L }eQ1#LISh޿Wac,w*ʻzM*m'MVx"2*S#9$.~ o>FOt}iQ0s2X, ΋hvaTVVb |jߨjDܦѧw|HA y~^oRҥK_ +0o]Yy]b%r*ZH!JO& eal5vsuc| ϺቬW'Ir}gmвΥha9`Ks-M佩S&PjDu~85'7gl}B8hgG += zЃ!et>mM8Gҡ1_*;hw'!,oO8F*/D$SuXtMp\-^)a3-2:k:uwyMz u${?):r웑|^OWLapMXFv6/f>kВsc>{y_sut:|6KTۙ>o rW~TOv+޲ᶶ>eltrT@~wxq/+1 ox>?{?_t)g y|ÊTs[\ 0?y\w(ʫNv6XCݢRכ̩koۮiԝeRVyO;rȸy9y+F}a#n6\vMyɖ2-<+$ `ߘh@gLΥ0:ԗa| gq#3:zC=V2->2Bi:uuՎ9j쮕bҺnt2e )OlSIm oeZNjl3rd2'yz5̏m26%MAHwaw:%|gu!3lVV, cgrnJ,#x\4[dGcI?g9YG4gb.Yfܯ"߯#6dy =ԭ:e1?s27'.R&tmkݺ(8_Oq:gMg8E%ѐF\O*g9.[bWMwjNGJo*zSs_'r7*'0ۉ/l/'92?Hն{f HlKR0vZ)ԯ>ԧV&aݒ-,gCUlXeq!e.wɲb![[rM-tb:שSk ҖRzڻ6uBF dS7=)I!~k+&'P^rSHZn[hJT^(SnЦ>I,C7F,կkr T]#cX/'ف{C~9 VPM'RwFv*]ahV'PUg⸙lg\Fdl6zzlZ")5 R^@n=v]S H5X]FӰ'*[ȡˊP 5]gw!dž/=95;9P|%^&[~,d :"@a!/02hm$[eirirs6ȝ2??N|Hy6 [Wޓ_ȃd؆MGBvy)54?}׼T ג!kܞ>ȾwLxxb d>W7þ^R}84Cri +2/jF}OtחtSK{Q79XB ^]6QRm.FVVG4[i펩j['j/S,Կ=zN VMYox-Zޜtv)bɬb_Qy)Iv6ۮ=/%2KpS]s-J՟ig8jvdL#0+$ݔƛ>q;Q8D69j/`}Fwj^Hoe +g9{33'reTQIЛπ$)iy5}2 b +endstream +endobj +464 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 463 0 R +/Resources 4 0 R +/Annots [ 465 0 R 466 0 R 467 0 R 468 0 R 469 0 R 470 0 R 471 0 R 472 0 R 473 0 R 474 0 R 475 0 R 476 0 R 477 0 R 478 0 R 479 0 R 480 0 R 481 0 R 482 0 R 483 0 R 484 0 R 485 0 R 486 0 R 487 0 R 488 0 R 489 0 R 490 0 R 491 0 R 492 0 R 493 0 R 494 0 R 495 0 R 496 0 R 497 0 R ] +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +465 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 799.173057 99.933071 780.423057 ] +/BS << +/W 0 +>> +/Dest (cb10-56) +>> +endobj +466 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 780.226036 99.933071 761.476036 ] +/BS << +/W 0 +>> +/Dest (cb10-57) +>> +endobj +467 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 761.279014 99.933071 742.529014 ] +/BS << +/W 0 +>> +/Dest (cb10-58) +>> +endobj +468 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 742.331993 99.933071 723.581993 ] +/BS << +/W 0 +>> +/Dest (cb10-59) +>> +endobj +469 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 723.384971 99.933071 704.634971 ] +/BS << +/W 0 +>> +/Dest (cb10-60) +>> +endobj +470 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 704.437950 99.933071 685.687950 ] +/BS << +/W 0 +>> +/Dest (cb10-61) +>> +endobj +471 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 685.490928 99.933071 666.740928 ] +/BS << +/W 0 +>> +/Dest (cb10-62) +>> +endobj +472 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 666.543907 99.933071 647.793907 ] +/BS << +/W 0 +>> +/Dest (cb10-63) +>> +endobj +473 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 647.596885 99.933071 628.846885 ] +/BS << +/W 0 +>> +/Dest (cb10-64) +>> +endobj +474 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 628.649864 99.933071 609.899864 ] +/BS << +/W 0 +>> +/Dest (cb10-65) +>> +endobj +475 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 609.702842 99.933071 590.952842 ] +/BS << +/W 0 +>> +/Dest (cb10-66) +>> +endobj +476 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 590.755821 99.933071 572.005821 ] +/BS << +/W 0 +>> +/Dest (cb10-67) +>> +endobj +477 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 571.808799 99.933071 553.058799 ] +/BS << +/W 0 +>> +/Dest (cb10-68) +>> +endobj +478 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 552.861778 99.933071 534.111778 ] +/BS << +/W 0 +>> +/Dest (cb10-69) +>> +endobj +479 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 533.914756 99.933071 515.164756 ] +/BS << +/W 0 +>> +/Dest (cb10-70) +>> +endobj +480 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 514.967735 99.933071 496.217735 ] +/BS << +/W 0 +>> +/Dest (cb10-71) +>> +endobj +481 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 496.020714 99.933071 477.270714 ] +/BS << +/W 0 +>> +/Dest (cb10-72) +>> +endobj +482 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 477.073692 99.933071 458.323692 ] +/BS << +/W 0 +>> +/Dest (cb10-73) +>> +endobj +483 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 458.126671 99.933071 439.376671 ] +/BS << +/W 0 +>> +/Dest (cb10-74) +>> +endobj +484 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 439.179649 99.933071 420.429649 ] +/BS << +/W 0 +>> +/Dest (cb10-75) +>> +endobj +485 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 420.232628 99.933071 401.482628 ] +/BS << +/W 0 +>> +/Dest (cb10-76) +>> +endobj +486 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 401.285606 99.933071 382.535606 ] +/BS << +/W 0 +>> +/Dest (cb10-77) +>> +endobj +487 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 382.338585 99.933071 363.588585 ] +/BS << +/W 0 +>> +/Dest (cb10-78) +>> +endobj +488 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 363.391563 99.933071 344.641563 ] +/BS << +/W 0 +>> +/Dest (cb10-79) +>> +endobj +489 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 344.444542 99.933071 325.694542 ] +/BS << +/W 0 +>> +/Dest (cb10-80) +>> +endobj +490 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 325.497520 99.933071 306.747520 ] +/BS << +/W 0 +>> +/Dest (cb10-81) +>> +endobj +491 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 306.550499 99.933071 287.800499 ] +/BS << +/W 0 +>> +/Dest (cb10-82) +>> +endobj +492 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 287.603477 99.933071 268.853477 ] +/BS << +/W 0 +>> +/Dest (cb10-83) +>> +endobj +493 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 268.656456 99.933071 249.906456 ] +/BS << +/W 0 +>> +/Dest (cb10-84) +>> +endobj +494 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 249.709434 99.933071 230.959434 ] +/BS << +/W 0 +>> +/Dest (cb10-85) +>> +endobj +495 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 230.762413 99.933071 212.012413 ] +/BS << +/W 0 +>> +/Dest (cb10-86) +>> +endobj +496 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 117.080284 99.933071 98.330284 ] +/BS << +/W 0 +>> +/Dest (cb10-87) +>> +endobj +497 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 98.133262 99.933071 79.383262 ] +/BS << +/W 0 +>> +/Dest (cb10-88) +>> +endobj +498 0 obj +<< +/Filter /FlateDecode +/Length 5118 +>> +stream +x]ߋ庑~?y\G?JR }XX >IX.!~dٮӖ{z ;LK*,}Jm{5Lv`)巋R03XP͑W4\ [՟/-u 9Gg_8%"cR Q)v%Ѷ( "AαR0S~[aK"asHJ5:c2bSl":=jejYBMN+~lZYb2-!,` yNZY->zke 8NV]ʲBf",pm3aʬ2( RLJ-фTи{k\h\AdոoبqLqzqJYN DqzqzFzM ok\@]oMҸFNF 1 ؐu .@-K׸.4 u8 E ch + z8[@ߣy p!,Ѹ wmS4jkç&2o˃1k\pT^}i\Ske]Ѹ w7 "иmU`2 Zr $jY$qx0C5.]qAkaZ[<; נ VVZxXV5&e "xWܦ10p@4.LCN)j\AB>,8$Aeei\z>  "EpQ-5. 4.H!"Cje @j& Rj\\j`Lz @K YҸ e؂q\Vfa l Z Dcdh\\]<5.`` Se4qGq$Hs2i\$d a3N`SͤqA6Ad @(3Fl# 0/%d 77)6(fҸ l Q2؂)+|줕E2i\c|5.@|$o&u2I+"2&"[-7 ސ,'+[%JꚁJ.LdUUZ./1p}]C$1-}9jAz|ݸRVZS:q F aktĈg#-hqZt#nb&b"v)X uV_C)o0ݮ +7Ty Kѐ|vz -*ۧ }6vÛ:י_/OO. ?ޟ}DNMMi4RO,rsypO;=1AY~kZ؏p݌l\>WG`.>Gߏc@b49 #6x5sCi)Y%"ۦ?V>1V25E=2 T, +Yrvةs4"KBqǥ'~)X,V1*'l +g}iwm0p*֣;-\e4vGO ȶuԱA'K4)+cB;uPdގ3ɗK#>O\t ʑvض}1 +=W=;m$Glښ42 >1-H .X"9j3YqBӀ4@Jخʍ5r}+7P籛gIg#,ڧs:YB9} xFk7~(ܶs$j :'ZT(J +z n:灢<ڝ /_G|rHP\ ;odݕƲP2xt-7=ht.Y攣q+hg֣G%``4ؽ9p'#t[ #߬c5p&&ں!MҴ{"q$k8bFIv!ݧsH3!4b D +捛,p>` piȑRִ{.$0+o&:灱C;:e?zfɶI(5{z9xztHAn" E2J{Dnvq#Ӂm[Gҁv7n$r_}|fكbyaD]9m&v@ ?1ysC͙f@V{Ml@:̲ŒDf}:hYٲё^y@l} bJ!FDzEKv[D{*:u1tC8HEݪR$͎n{ңEy¦h$ ;u3n5F?㇧&jb +7ERE}YڛT݋[,~l-~(=H s'=N@Xv +~z=vWSS2pSm޷->-ko,ܟufЃu#-[߶mY6j?-9Mq:灤]$ӿ6ݩsfyG kK#>~mzkϑߵiސC@6OcPi +m0?r?ʊ!Q܃ײIuĉ+Vg|O8-vuR.pwV<MgA=\tg oШb`G7_Mނ8]iD3G)?$Ͽ|r٪z$*(Yĝz%$0D&S=r_~d$3Nui5Tꄦx-č|yݿ]] ;~1ϲ 1_ʷ4xԽv{}9~2ZKȇdTq`~Xp\0ofv"_#EPuE8jdSJv!k0\OCՅr5|Ag(G{ÊnrsXȃb5%b3 ;pV[oa +M,CXMo6_ KYʷʩJۊ7F* f㤈nE) 7&_ j\)E7-ԑJϪl uϵG4g`9b|u)2e 7q"}lϗa + +Zo[lfR" S{!*B +XoGs#27xITm0_ѱZ,,h6g@ +xQ\fke'y:O2.n z+#l#{' B_p~ܙrBz2z-wkEXM#t,<  Տ;5f&RVu7=^VXa}8yi|:Xrps]9%QJQA{R6 ٲȶb?Cᥭ #e4E6^ES!om {_={'c"^ϭ^H+R?a9~|'KssfSc-hqb6UmTdFt)zTlZYTcYC;*C;*qSW[<^| B 4_kXYl暾ȶmӲmelH#S3MWk]gZ* '&/$!$rM͒a8+em 1&Y (ͩjrI:@Y™ilVpzϴS!]5d3.'v +endstream +endobj +499 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 498 0 R +/Resources 4 0 R +/Annots [ 500 0 R 501 0 R 502 0 R 503 0 R 504 0 R 505 0 R 506 0 R 507 0 R 508 0 R 509 0 R 510 0 R 511 0 R 512 0 R 513 0 R 514 0 R 515 0 R 516 0 R 517 0 R 518 0 R 519 0 R 520 0 R 521 0 R 522 0 R 523 0 R 524 0 R 525 0 R 526 0 R 527 0 R 528 0 R 529 0 R 530 0 R 531 0 R 532 0 R ] +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +500 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 799.173057 99.933071 780.423057 ] +/BS << +/W 0 +>> +/Dest (cb10-89) +>> +endobj +501 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 780.226036 99.933071 761.476036 ] +/BS << +/W 0 +>> +/Dest (cb10-90) +>> +endobj +502 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 761.279014 99.933071 742.529014 ] +/BS << +/W 0 +>> +/Dest (cb10-91) +>> +endobj +503 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 742.331993 99.933071 723.581993 ] +/BS << +/W 0 +>> +/Dest (cb10-92) +>> +endobj +504 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 723.384971 99.933071 704.634971 ] +/BS << +/W 0 +>> +/Dest (cb10-93) +>> +endobj +505 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 704.437950 99.933071 685.687950 ] +/BS << +/W 0 +>> +/Dest (cb10-94) +>> +endobj +506 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 685.490928 99.933071 666.740928 ] +/BS << +/W 0 +>> +/Dest (cb10-95) +>> +endobj +507 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 666.543907 99.933071 647.793907 ] +/BS << +/W 0 +>> +/Dest (cb10-96) +>> +endobj +508 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 647.596885 99.933071 628.846885 ] +/BS << +/W 0 +>> +/Dest (cb10-97) +>> +endobj +509 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 628.649864 99.933071 609.899864 ] +/BS << +/W 0 +>> +/Dest (cb10-98) +>> +endobj +510 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 609.702842 99.933071 590.952842 ] +/BS << +/W 0 +>> +/Dest (cb10-99) +>> +endobj +511 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 590.755821 99.933071 572.005821 ] +/BS << +/W 0 +>> +/Dest (cb10-100) +>> +endobj +512 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 571.808799 99.933071 553.058799 ] +/BS << +/W 0 +>> +/Dest (cb10-101) +>> +endobj +513 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 552.861778 99.933071 534.111778 ] +/BS << +/W 0 +>> +/Dest (cb10-102) +>> +endobj +514 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 533.914756 99.933071 515.164756 ] +/BS << +/W 0 +>> +/Dest (cb10-103) +>> +endobj +515 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 514.967735 99.933071 496.217735 ] +/BS << +/W 0 +>> +/Dest (cb10-104) +>> +endobj +516 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 496.020714 99.933071 477.270714 ] +/BS << +/W 0 +>> +/Dest (cb10-105) +>> +endobj +517 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 477.073692 99.933071 458.323692 ] +/BS << +/W 0 +>> +/Dest (cb10-106) +>> +endobj +518 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 458.126671 99.933071 439.376671 ] +/BS << +/W 0 +>> +/Dest (cb10-107) +>> +endobj +519 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 439.179649 99.933071 420.429649 ] +/BS << +/W 0 +>> +/Dest (cb10-108) +>> +endobj +520 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 420.232628 99.933071 401.482628 ] +/BS << +/W 0 +>> +/Dest (cb10-109) +>> +endobj +521 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 401.285606 99.933071 382.535606 ] +/BS << +/W 0 +>> +/Dest (cb10-110) +>> +endobj +522 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 382.338585 99.933071 363.588585 ] +/BS << +/W 0 +>> +/Dest (cb10-111) +>> +endobj +523 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 363.391563 99.933071 344.641563 ] +/BS << +/W 0 +>> +/Dest (cb10-112) +>> +endobj +524 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 344.444542 99.933071 325.694542 ] +/BS << +/W 0 +>> +/Dest (cb10-113) +>> +endobj +525 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 325.497520 99.933071 306.747520 ] +/BS << +/W 0 +>> +/Dest (cb10-114) +>> +endobj +526 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 306.550499 99.933071 287.800499 ] +/BS << +/W 0 +>> +/Dest (cb10-115) +>> +endobj +527 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 287.603477 99.933071 268.853477 ] +/BS << +/W 0 +>> +/Dest (cb10-116) +>> +endobj +528 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 173.921348 99.933071 155.171348 ] +/BS << +/W 0 +>> +/Dest (cb10-117) +>> +endobj +529 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 154.974327 99.933071 136.224327 ] +/BS << +/W 0 +>> +/Dest (cb10-118) +>> +endobj +530 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 136.027305 99.933071 117.277305 ] +/BS << +/W 0 +>> +/Dest (cb10-119) +>> +endobj +531 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 117.080284 99.933071 98.330284 ] +/BS << +/W 0 +>> +/Dest (cb10-120) +>> +endobj +532 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 98.133262 99.933071 79.383262 ] +/BS << +/W 0 +>> +/Dest (cb10-121) +>> +endobj +533 0 obj +<< +/Filter /FlateDecode +/Length 5065 +>> +stream +x][,~_Qφu Â1p]8CcB[fFUfxJ.!EI6W ~$2*]o?]~hƿcPwW)*orH'b6?6sus*`G&S1Ũ@ i*F@ +F9K$#Yړ"ؠDdmH l8IlhH" }HFci`D9+ЌW$5*lHh%s*D2HB>I4/>k#X `Q +8#%, s() Xr&9 댎pJX*alJZڤI ҢtpztA]뵕l?Z/+a=hIXʔ["a 'i2) OQ JH‚1D Vi 4%,LJK IX3xTH% \'[x#aAޠ&DAKX XC-Hs$,H` R4`KX&^' Rn& R/V[ oF /$,:(v%,$ě$,x3IXFO7" ,[`MD f fnMx@x3]L 4HXěl^x}&GI\1Ѐ^*.h`/U 4c$ql"LA\5Yhq@էy*@"lfɌiA{1;ٖR־ Q\R!8<,s ~ 3`3 S sY ggv)w$^c] S4~V@.-% " +M/qap~/?zv|_>0C^TCoFL7&7|׏|OSVw<> +*)%q4u\QK8h67/1rhy7=s,>2#Z C c?ް xiPLKYB^F0V9RY F!6C&f+>{7sG߄l%̒Փ7"Q!#X\QOpTUE@/Sk6*g՝\V˒ۤ$)}$x}<P( Q*}bB|Eu:T`M9*)T\Q'ܱ6tb*:/4`]j&IcCmp2l>2#z !Yx 7B4RTAWqV<[KՔn=F~Ym\y[d7n]80zs&je,e/>crʿ,~.rdž(dQn[{pܫF~Mn!RWn~ =5!(ftLw/8o\QD +2wFMaRWET#oU`\XOw[T">۴ɝm/}&yWjLQNi[޳Xd?|ӴZƞ_~g^*\TMaQǚ3|fakkuҦ зjK/-)sOZ.{Xݵsaja%` pZe>bSG &ٗcmdždmwPTGAgik%_))yB4f4?%_i'g~1O񻍾OK_ N}nE?ƠF峭9Ʈ<_&nȍ"=o%-`켖t+GZ˩gʓM啗͗HF{csjҐ /To½N}i[ n}{8ӵ;/Xc;?ЏI?(Sǫɂ^Gzim\W5uKÇq۔+1s[<\EL&{9s+,Z7>̞em1gm[yɽ:YF3|}0'C>^rۆvO2Xw4Kф6 =Z.I0|S2#>m')^_Wu^J.;LD}Me 5al+`VsEG19F&z>o8$ny MBwیseFI1H䣽#u#_;(ϯ7> Yޝ/>Y,<" N>x*l\m皙G/ȯ]'QCVLy[hΕyx0 *Esf .N.cO6w_޷Ӿ%yI7%|/KKBy>i7ר\Jںu*IH^S[fmXܴs1{h7 _db;XLsegJKV|cүпlDpֽ}WIIw[Wݨ_eIo Os[n('oQ'-sjg_ +5xDTeB;" lH6dYd|:M'kܜ69e\t2O#ް(Mރ:LTR/]'ג2>yt|Eq`)WGx=Jm%8 r Ew76[nϗшt9&7ڶkhgMcp| B,S.E{q&SxF⨐漣Q0[F'꩹̷#]HCg&pͺmꛘʗZT.'!vӭj0;})|n1_wIŞ+`e:Ә iHb9dhiK].⣐оޜz{rC_F*:T-/ TS>*ŗ͞9=t!i őw.cyLB~ݞ*u)=TuZדtuc͍dUm<}z fRwɡcۜ N:oŰ6U?FF2X5M1H0Vlį[cIv9e/58ޑsŎ[vl]%ϖO'ا͔hil6t;n4-j:uNc dU7xیbwd~l^]C׮P0wi0שJ~Jzޒ9鑳IH sJ#kiivש4 3գ}d,e6RUV0LwPκt +B||4-i5JF[[89?%B>&LU?%Z@X0w+8/_]鏑CE?|3%{QqI^W6/ +endstream +endobj +534 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 533 0 R +/Resources 4 0 R +/Annots [ 535 0 R 536 0 R 537 0 R 538 0 R 539 0 R 540 0 R 541 0 R 542 0 R 543 0 R 544 0 R 545 0 R 546 0 R 547 0 R 548 0 R 549 0 R 550 0 R 551 0 R 552 0 R 553 0 R 554 0 R 555 0 R ] +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +535 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 799.173057 99.933071 780.423057 ] +/BS << +/W 0 +>> +/Dest (cb10-122) +>> +endobj +536 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 685.490928 99.933071 666.740928 ] +/BS << +/W 0 +>> +/Dest (cb10-123) +>> +endobj +537 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 666.543907 99.933071 647.793907 ] +/BS << +/W 0 +>> +/Dest (cb10-124) +>> +endobj +538 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 647.596885 99.933071 628.846885 ] +/BS << +/W 0 +>> +/Dest (cb10-125) +>> +endobj +539 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 628.649864 99.933071 609.899864 ] +/BS << +/W 0 +>> +/Dest (cb10-126) +>> +endobj +540 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 609.702842 99.933071 590.952842 ] +/BS << +/W 0 +>> +/Dest (cb10-127) +>> +endobj +541 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 590.755821 99.933071 572.005821 ] +/BS << +/W 0 +>> +/Dest (cb10-128) +>> +endobj +542 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 571.808799 99.933071 553.058799 ] +/BS << +/W 0 +>> +/Dest (cb10-129) +>> +endobj +543 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 329.565778 101.433071 310.815778 ] +/BS << +/W 0 +>> +/Dest (cb11-1) +>> +endobj +544 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 310.618756 99.933071 291.868756 ] +/BS << +/W 0 +>> +/Dest (cb11-2) +>> +endobj +545 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 291.671735 99.933071 272.921735 ] +/BS << +/W 0 +>> +/Dest (cb11-3) +>> +endobj +546 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 272.724714 99.933071 253.974714 ] +/BS << +/W 0 +>> +/Dest (cb11-4) +>> +endobj +547 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 253.777692 99.933071 235.027692 ] +/BS << +/W 0 +>> +/Dest (cb11-5) +>> +endobj +548 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 234.830671 99.933071 216.080671 ] +/BS << +/W 0 +>> +/Dest (cb11-6) +>> +endobj +549 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 215.883649 99.933071 197.133649 ] +/BS << +/W 0 +>> +/Dest (cb11-7) +>> +endobj +550 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 196.936628 99.933071 178.186628 ] +/BS << +/W 0 +>> +/Dest (cb11-8) +>> +endobj +551 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 177.989606 99.933071 159.239606 ] +/BS << +/W 0 +>> +/Dest (cb11-9) +>> +endobj +552 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 159.042585 99.933071 140.292585 ] +/BS << +/W 0 +>> +/Dest (cb11-10) +>> +endobj +553 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 140.095563 99.933071 121.345563 ] +/BS << +/W 0 +>> +/Dest (cb11-11) +>> +endobj +554 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 121.148542 99.933071 102.398542 ] +/BS << +/W 0 +>> +/Dest (cb11-12) +>> +endobj +555 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 102.201520 99.933071 83.451520 ] +/BS << +/W 0 +>> +/Dest (cb11-13) +>> +endobj +556 0 obj +<< +/Filter /FlateDecode +/Length 4901 +>> +stream +x]]$IѾ_¦_ 00xqףYa¿ՕUU}ۊ3tWD~DeD>l+2;sIϗO?Iϔb0)d+1BdR > ooH0p湁>Xaw=a  w!A V{ ɂY*pb XB%kXcK5[gf 2G#&i)UdLhX`;hXH֡dΪȀsl*af*ͤJxjŀ6 ?!*dـ+~u,!)*8+"/~ `E_}:꯳qq=O~z=}ZBorw>Og~('%_nc$bEqAs:|/#yߞ~i #Y?l&/yUfDNfXJcFyN}HH֙\zHJ煎eySIx }j>vFjqa_2ZT6Ŧ Ͳ,@d 'MlvoxYFK=dwazwÈoV1vϲyh{.T{y#H=;qTZ@$bt{]*ZJv @A]W68l{6醟ʢ)h y4VY9ϠGe,fnl׆T ,l]psT[O9e=eW @zkdZk .1(:Xݷ~ە(-<^NC|}FLI Ji^jz7V>,t8FI9+5ɻHIS 546sY +4fZ5}}Y??헫"+24Y0wI͟AIfH0}u$ߥ<ޥ;{4_r[~>iThB?MRr0m{os;ww`LRng<<0\;|wckgQhqI\}8]^Riq6{/>A&jos$9{(bxv0atƙI-jBƎ꜓cCѥ0g8m7ԋ!є_4V~Ypeik\60-A+Atr;8yT4ϱ8gg\䦓Dyw <ϣ\ї݇RoE4q;y)oN?Y)YRzhȝ4ſm[ǽmh.ȣ'Cغ@<~'ӆj ͏{?5{M@teeՊ-Hd/V0sQwHy{ h{Ԃ5zhFja+mQiu .@JLU8g2qV[)S2t1ZRnx-ZќS;SX2+8nT)Ҷumkp$la]sͳ-Rտ 6j8jv> +endobj +558 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 799.173057 99.933071 780.423057 ] +/BS << +/W 0 +>> +/Dest (cb11-14) +>> +endobj +559 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 780.226036 99.933071 761.476036 ] +/BS << +/W 0 +>> +/Dest (cb11-15) +>> +endobj +560 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 761.279014 99.933071 742.529014 ] +/BS << +/W 0 +>> +/Dest (cb11-16) +>> +endobj +561 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 742.331993 99.933071 723.581993 ] +/BS << +/W 0 +>> +/Dest (cb11-17) +>> +endobj +562 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 723.384971 99.933071 704.634971 ] +/BS << +/W 0 +>> +/Dest (cb11-18) +>> +endobj +563 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 704.437950 99.933071 685.687950 ] +/BS << +/W 0 +>> +/Dest (cb11-19) +>> +endobj +564 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 685.490928 99.933071 666.740928 ] +/BS << +/W 0 +>> +/Dest (cb11-20) +>> +endobj +565 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 666.543907 99.933071 647.793907 ] +/BS << +/W 0 +>> +/Dest (cb11-21) +>> +endobj +566 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 647.596885 99.933071 628.846885 ] +/BS << +/W 0 +>> +/Dest (cb11-22) +>> +endobj +567 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 628.649864 99.933071 609.899864 ] +/BS << +/W 0 +>> +/Dest (cb11-23) +>> +endobj +568 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 609.702842 99.933071 590.952842 ] +/BS << +/W 0 +>> +/Dest (cb11-24) +>> +endobj +569 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 590.755821 99.933071 572.005821 ] +/BS << +/W 0 +>> +/Dest (cb11-25) +>> +endobj +570 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 571.808799 99.933071 553.058799 ] +/BS << +/W 0 +>> +/Dest (cb11-26) +>> +endobj +571 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 101.433071 444.418756 101.433071 425.668756 ] +/BS << +/W 0 +>> +/Dest (cb12-1) +>> +endobj +572 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 425.471735 99.933071 406.721735 ] +/BS << +/W 0 +>> +/Dest (cb12-2) +>> +endobj +573 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 406.524714 99.933071 387.774714 ] +/BS << +/W 0 +>> +/Dest (cb12-3) +>> +endobj +574 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 387.577692 99.933071 368.827692 ] +/BS << +/W 0 +>> +/Dest (cb12-4) +>> +endobj +575 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 368.630671 99.933071 349.880671 ] +/BS << +/W 0 +>> +/Dest (cb12-5) +>> +endobj +576 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 349.683649 99.933071 330.933649 ] +/BS << +/W 0 +>> +/Dest (cb12-6) +>> +endobj +577 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 330.736628 99.933071 311.986628 ] +/BS << +/W 0 +>> +/Dest (cb12-7) +>> +endobj +578 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 311.789606 99.933071 293.039606 ] +/BS << +/W 0 +>> +/Dest (cb12-8) +>> +endobj +579 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 292.842585 99.933071 274.092585 ] +/BS << +/W 0 +>> +/Dest (cb12-9) +>> +endobj +580 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 273.895563 99.933071 255.145563 ] +/BS << +/W 0 +>> +/Dest (cb12-10) +>> +endobj +581 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 254.948542 99.933071 236.198542 ] +/BS << +/W 0 +>> +/Dest (cb12-11) +>> +endobj +582 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 236.001520 99.933071 217.251520 ] +/BS << +/W 0 +>> +/Dest (cb12-12) +>> +endobj +583 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 217.054499 99.933071 198.304499 ] +/BS << +/W 0 +>> +/Dest (cb12-13) +>> +endobj +584 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 198.107477 99.933071 179.357477 ] +/BS << +/W 0 +>> +/Dest (cb12-14) +>> +endobj +585 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 179.160456 99.933071 160.410456 ] +/BS << +/W 0 +>> +/Dest (cb12-15) +>> +endobj +586 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 160.213434 99.933071 141.463434 ] +/BS << +/W 0 +>> +/Dest (cb12-16) +>> +endobj +587 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 141.266413 99.933071 122.516413 ] +/BS << +/W 0 +>> +/Dest (cb12-17) +>> +endobj +588 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 122.319391 99.933071 103.569391 ] +/BS << +/W 0 +>> +/Dest (cb12-18) +>> +endobj +589 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 103.372370 99.933071 84.622370 ] +/BS << +/W 0 +>> +/Dest (cb12-19) +>> +endobj +590 0 obj +<< +/Filter /FlateDecode +/Length 3157 +>> +stream +x͛[% ϧ8a`00$kfvWKήװ;̞n.*կ:=|%|*Z[z}|B$¿fSߒLZBk7PZ%_.t5rtGSRKTR +ĔUGlST4꽪 m5y*1VGUahaW5 RqTL#!*q +=]kDO'rQNZ9FlIJ2l,`KE<]-]-` ebO$VU]]jH)*B9$:(LR..–L:ؒآtDRZn!aO`KE]l{Pqc@,%hA V Zz-E< ĖV<1t=#0lQ\W[<@@[x,Vo8{,fqfX @.\y,0WNbJc5{,pWc"` cx,PpWwjfk% +J@Xc1܍wrpI~8[XcAwas4h \1}.qgb<$px,H.Y=;jX]ԞcAw7$p<` Lp]WZ=$pX  + @C=dp{X]J" 2Kcl5`5P)<Kt.Qi 2kۦǂ\LGjhdt\]O +A,W[֡I J-ǂa BBcDnz[.n~,Ԧ걠| r ׮BcA)Rb{,K `t J-X* {:-= ҅cAѠǂ+L C#P{:%3ǂ9X&H` 0cvX&W[$ǂZcA%RXߴ|x,h[y,h q<47z,|z,hB79I|x,h +[BW[45g݂I7A'OaKl걠lfX0&"zHZYp75ӁL DdWV8{b@@o؋ݪgZLd5Vdr EGTPgbٰ,9f1y|ן._{鿗wNE%ɖHln*l+B5 a1#JF2؊ݶɰlf3ۓ/2TIm:Vb!}@Lv+rs1#|KёEc'^C?=\o'vbp#]_~>DWl?ɼg'z~;Z{Zg^~s/zi/GF??7>|{q+B> ep {ʼn]Oc"C6<rHd7ڲ$'WXO(Ü)s_nTS?u7熯pEk9> ;YZsz_"¤rضXE]/laUb9d>m;N(sЈX]?4B RȨs@JJOz^='RctxZ- G9V~9{}>WyS|Rg ,RH[CΗžIpCcm c$ɎrtC'أrr9֞ Q6K^vb m38Nº~88ɽJpptùtѝm|ζsr߾ŮlAUqV8~D loS9d$'(>mutӶ7wK"%a(%&Ot?gkO"TT7 aT3.Z#>n}^a'8ۻbiX[Blgtp|L +endstream +endobj +591 0 obj +<< +/Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 595.275591 841.889764 ] +/Contents 590 0 R +/Resources 4 0 R +/Annots [ 592 0 R 593 0 R 594 0 R 595 0 R ] +/TrimBox [ 0 0 595.275591 841.889764 ] +/BleedBox [ 0 0 595.275591 841.889764 ] +>> +endobj +592 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 780.226036 99.933071 761.476036 ] +/BS << +/W 0 +>> +/Dest (cb12-20) +>> +endobj +593 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 742.331993 99.933071 723.581993 ] +/BS << +/W 0 +>> +/Dest (cb12-21) +>> +endobj +594 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 723.384971 99.933071 704.634971 ] +/BS << +/W 0 +>> +/Dest (cb12-22) +>> +endobj +595 0 obj +<< +/Type /Annot +/Subtype /Link +/Rect [ 99.933071 704.437950 99.933071 685.687950 ] +/BS << +/W 0 +>> +/Dest (cb12-23) +>> +endobj +596 0 obj +<< +/Title (Hardening Guide with CIS v1.6 Benchmark) +/Dest [ 6 0 R /XYZ 85.009843 502.818898 0 ] +/Count 14 +/First 597 0 R +/Last 608 0 R +/Parent 611 0 R +>> +endobj +597 0 obj +<< +/Title (Overview) +/Dest [ 30 0 R /XYZ 84.259843 272.631496 0 ] +/Count 1 +/First 598 0 R +/Last 598 0 R +/Parent 596 0 R +/Next 599 0 R +>> +endobj +598 0 obj +<< +/Title (Known Issues) +/Dest [ 45 0 R /XYZ 84.259843 799.370079 0 ] +/Count 0 +/Parent 597 0 R +>> +endobj +599 0 obj +<< +/Title (Configure Kernel Runtime Parameters) +/Dest [ 45 0 R /XYZ 84.259843 539.222079 0 ] +/Count 0 +/Prev 597 0 R +/Parent 596 0 R +/Next 600 0 R +>> +endobj +600 0 obj +<< +/Title (Configure etcd user and group) +/Dest [ 45 0 R /XYZ 84.259843 283.686971 0 ] +/Count 1 +/Prev 599 0 R +/First 601 0 R +/Last 601 0 R +/Parent 596 0 R +/Next 602 0 R +>> +endobj +601 0 obj +<< +/Title (Create etcd user and group) +/Dest [ 45 0 R /XYZ 84.259843 169.788971 0 ] +/Count 0 +/Parent 600 0 R +>> +endobj +602 0 obj +<< +/Title (Configure default Service Account) +/Dest [ 53 0 R /XYZ 84.259843 561.590928 0 ] +/Count 1 +/Prev 600 0 R +/First 603 0 R +/Last 603 0 R +/Parent 596 0 R +/Next 604 0 R +>> +endobj +603 0 obj +<< +/Title (Set automountServiceAccountToken to false for default service accounts) +/Dest [ 53 0 R /XYZ 84.259843 522.092928 0 ] +/Count 0 +/Parent 602 0 R +>> +endobj +604 0 obj +<< +/Title (Configure Network Policy) +/Dest [ 67 0 R /XYZ 84.259843 599.840928 0 ] +/Count 1 +/Prev 602 0 R +/First 605 0 R +/Last 605 0 R +/Parent 596 0 R +/Next 606 0 R +>> +endobj +605 0 obj +<< +/Title (Ensure that all Namespaces have Network Policies defined) +/Dest [ 67 0 R /XYZ 84.259843 564.092928 0 ] +/Count 0 +/Parent 604 0 R +>> +endobj +606 0 obj +<< +/Title (Reference Hardened RKE cluster.yml Configuration) +/Dest [ 77 0 R /XYZ 84.259843 191.441066 0 ] +/Count 0 +/Prev 604 0 R +/Parent 596 0 R +/Next 607 0 R +>> +endobj +607 0 obj +<< +/Title (Reference Hardened RKE Template Configuration) +/Dest [ 390 0 R /XYZ 84.259843 518.211778 0 ] +/Count 0 +/Prev 606 0 R +/Parent 596 0 R +/Next 608 0 R +>> +endobj +608 0 obj +<< +/Title (Reference Hardened cloud-config Configuration) +/Dest [ 534 0 R /XYZ 84.259843 537.158799 0 ] +/Count 2 +/Prev 607 0 R +/First 609 0 R +/Last 610 0 R +/Parent 596 0 R +>> +endobj +609 0 obj +<< +/Title (Reference Hardened cloud-config for SUSE Linux Enterprise Server 15 \(SLES 15\) and openSUSE Leap 15) +/Dest [ 534 0 R /XYZ 84.259843 397.010799 0 ] +/Count 0 +/Parent 608 0 R +/Next 610 0 R +>> +endobj +610 0 obj +<< +/Title (Reference Hardened cloud-config for Red Hat Enterprise Linux 8 \(RHEL 8\) and Ubuntu 20.04 LTS) +/Dest [ 557 0 R /XYZ 84.259843 511.863778 0 ] +/Count 0 +/Prev 609 0 R +/Parent 608 0 R +>> +endobj +611 0 obj +<< +/Count 15 +/First 596 0 R +/Last 596 0 R +>> +endobj +612 0 obj +<< +/Length1 9900 +/Filter /FlateDecode +/Length 6559 +>> +stream +x:{\UU^py? 8Tt|:)HZj +>2%S35#sdy5Ԛ;cnh6ך{0m{݋z|[{}lU2$g>6g694<4mH+?uh/@/бL= GfLMZNCc{'W`^ꏟ67g=iop~8 I0E/>w^]R*E}97DnGgfe熝߾3hzRI|GϘ;'H͙S8-;zs+WK&d͋~q0]**,!L=|Qq^@÷Ԕ d'MpG}}a*h׷Uj?xC⤮ +_0WP}QcL8@/ 6];':fꆶ?/9hYATeI ES8\v5 X'mڬ|;!UO]6O#&ׁowQ)N;$ GP? +1 +U0@MNo i}!c!`&̆BXԦisv0 gt mmmW^[:bP:=<3zB;#zQ7f:޶ rXǃЭO={̘^:$됬4;Zi|H>mZ7<.|ڌlOE xʖ7'xrOmNO7G"wo w7ŊM7~ԃ7jz 65o_kHkzYoT~r5Rr (xP~y)^/ +] +|_{/ssn"xzƷE o |SqolxLQ;Wy@c+)1ȷs[l@b&$pc(iQu nXoz+>ی]3=k5ϸ)#?Ƨ^Ջ>El> V4*'4c% TbI+\Ire=m^Zҥ$K3]ԍ \,1+.B.8K5cBDl8S8: +&0G`jƩ"pI'N08Lc 28q?>Z`!p#6>J#6|XH)ppC|pkpH |Hj*'?ȗA} O/S__ľf׆fI>x' +ό <{WO#eÞF쑈<>9xw79[w{c6_t1Z`H_ >#aJ,b5cH@]r1$ ПKS@8ډW`-E\ +sYɆF* NTrQI,4K=2/]4p5’ 稜#̣Z 4rcURO 7"AV9+I4 +pKpe!‘œ,@$5<%9K.aIN)9 aePp$ld+|p/|NhY9*vi'+2TC#gB/m"Qw= a0._{x32:| @@ 23ir:{z es&TEhN&e=E +9J#5,XChk|xJ]Z$ge!okў1\^؈lVl\0{z`XJ\0ZpBO5IWǿ[O5i{iu7|֚(92N9ӅYĎ}_Fe2$۩,_Wh* %U/ nCalJʥGIBtnJE!rTy &x\Vj5aJ0B3.͜PON#It)r/Ҕ~vNG"!6wو{aa͗Rη\C6chSJm.[*r]j%aAn"VS}`kh#%$d xA>)C)l +T\Nľ}b+&&*R¾}儨p "_ڹrt3=E߁gʸh7X z9xt ǎ㨴#"_cO, +ղX& +vLDBXxU^ U/05؟2BC [HMĤFJAHF,#t4D˩"zϿ(Xu6$_k._RKܼtUWHyEzj3ܦw> 5]層@^:`}uWC$8m]5t}#oPXDiRd~.E#VmY`ٵYחW㬂s.?:UWę^6mg }hUHbT:^pU[שC%mD[Kc˩;ԉK7)) n{Dr];T>)KZC,`ge!3?昽2[l8el?<%*~'sk>tmZ-"&ŀliWl"tP#5fB-I(ξ+w[#[Jҙhn^GUICG>;Φyާ/$pY|  ;a%|J$XgEаP(LRXX`94Lv1pХ.W7)[dȠ@kOC3SM-Mmj4:5;#z"Dƀp#"cb;|366&&OtR<^_ >f}S^TJCe|_޿Kº|k퍲Ig 3rRoW WX Us!1jBtZ 2)-$$&%'D$DzB<0O'["2222bE؛K;u. + + +ϊ( +- ++ +/(-+/rOKgJ&$IM"AbH™tOnzSV?zI/m:w|ҿߔKsJ{'vlU^{cc[4v!llYi^-8|`s `nؾ1قʃqv:AdG0kx3Ȯ)'S퐓_pj|h6 4HTɓ' {փYV/ۈ2P<$tvTyCSMS7gֆ<:&m< LxmɹsK^k=ҳ/ai6fiL-\];G7tUFrByŌJzz ;֥ᵬrIKwHN)rv\OԐڐ߆ ᩐRTWja60B!+ +]A)4Gߑ,3B7e--:3-qŵ| ^i-Vig9нLx6'1^D<3btTb}L7XFא@P-C#|HTO5 ^\8J!w/^]~!&w$ ]vP+Dsѣ͡Kkiyot;}Ι>Z|& + }G7ޜ<- `d6 g^͒K6}ݢY-( xώJ[o?S*]?Mܣq?x +vb&q큒 +(%7)`ĸ t'gYYʜ ]Fha,T**riSRh$+vUqr ltHgq·)u3|.Kqe`UXE1BQڂM|ԩomS(GMeSO_5Y-v-Ih< $da_{q +78Ǚ&ifb7NSU&8 A,Mt)$2YgϜ<ΡU~Zen&W6B>ڰh? ~RwMMm\y)4\Mdi2g0H<ߴtJio05H = CdDE6sj6_ʃ 첸͢XFcP3M I0Og) ltKu,`[^orlgiZiJfIڗ>HN ޖ@emk\͡ V!("X 0 N8z&B$jD8 &PBw1A6̅4: +9Ta\%z/yf!չik&~, hlZFYn<, icWd&$pZ_Hgs?3VRB;̦Qm-1%ItܽsMyh?yIڗ6RAe6c‡/v*Le;j n;錑tRFA: z-yT/.˧R{zqlX>#`flX>'0\͞?F +gυ +r~~tghϻegdڠ\8 _nw_}6=g(Jx k]8tWYڍ4 P +endstream +endobj +613 0 obj +<< +/Length1 6944 +/Filter /FlateDecode +/Length 4153 +>> +stream +xYyxSו>Y2/l%['ɻ,[vWɋŘLiI&MB)kMJ/4!Mffi&I6i:COsmBso=w߽{s9OF!!LEBQmr6+3Ij]ޘ-ΑDB'ۚ=M^6(61{`(Gii՗`G.Cu*OY&OgZ}O ^FJ9:.,,xB {A9ZT^n2!DqGP@軁sJ|e< eB_0 +R }6kG~3i,BD1z@4BOLj5Κ}u^ߛ>(P Zϯ:XHbIJ!Ch Y&(1HJQـ,=?.rQ#@<* /=/t +xzgCd%!Gc(Don1X̯x҉I1}A,!Ў?QDQo&)h!B oQafJ4c&:8vZAaDB-`Xg-JdbTFWF|Qdl"O +:v[WZ]j ܇)tGd܀S6aIΜ~>Wp_svSF&fpLiaN⭷`gOCYP$QRY#$b%o_{]o΍cOxϼv:L)̬ϔ5ajQ0q%zzgu}{8Q-g('?ۛmCj6ď'nhViXe8B(+ˣd˄(9-th&r¦Ic8YP(dZz{p/> 1 &dЛ$dk<*F֑Vj!lS. j6TwMS 1bQ0 )DZr4BT,U?Hc`c*cͭ# [}j\[wcp=COP|@HWT˫@ԬF3O?bhdMvU"hij,uvQiKq!AKj.UOKݒa˳LZ~ǐJr͇R՘ e2Lkp]WSSn|;N^t02[ ~ KŚi*% Òj-ASھ}QUSWT\\. ++/ףwe:uuezFyhɧr,,&1S/*#%E`/*ě׺I&b;=(. >3tGqOGc2kVQ Cw;4Xj)}Q(<<5HvRwV|,daJ\䌄*MP(uOk{SE7dZؘ_;>uTm,MVh2*u\ZׇM3䧝&/QEQPF{rসgBTW,ZG7d۳6?ܗivN;FGinw_=V]a+͖Jzr +Z +E4'> dy z#+paɴwKGG'/Mnz20~ֈB2X-4tC,̉մMMJ}*=3~t^aQZשRDRR[VgM}Kߢ{M~M^G+驣Hds\F*/}ke27hë~A7>ݜc0lq:9M]#-? e> +D$e'N_w?ػktL= Sn۾o*Vo@ ~0\L%Px +n7&=F<gK;nȄVMk[osc/"s 5+-!~Ҳsv?9yV?뷔nol:@8$BaIZ\ 7&ŽA& 43hq%ekiy4 ac =s{z빣ܱ?9!;Z+b hIe8%}.n"S3S:xWh5|>Bg`kxp݇(rV O:Jqim=*2:[x5e5_`@ EF^,kBVodæ?ܵ ?>}H&eo0R2 Ӈ=KpZ-]x%wLhܙ0*ndk N:h'c=y"LۆP%ō.- yǐvCÖ kv9>'=YYRe( r+Jdg㒗- -c@ؤ۠Z3GƷϹ^KzX@J w`Ӟ72[`<+32+˰B$}X!p}v+ϕ+~:A /0F11n92|xˡ'9)SvN[$44P$?Q7 [Qnrɲ'e,T*,[m[ڤ-9lwJ[SR-xӼfe1=7xљo`&rA`E))ϵƭNv6\>t'dk)25&xOHaq⬠; G T +e3Ml6CB-{%a ?d35kpyK1t۟>扄1Rp֠fsfPpѰĢ|W㐕%ĢF㶲|ðdSTR֓|t3K#~e2յg ˜_)e$g+6]Zm~9G3٨ϏiD7eT䛜rwDI:rg*~3]yQ|6+C@?:|v45@6? ~y!>2'>E~{,|2ZX8jUCR2۹-?l8ѝNHp Jxl]5I~Ĭi@-]vGu9Mw퉺}bVbY_V> rV.9-|_@.@o + +dSS+OSeJwWbr)2uRUyʵD\D^g\.WY.,YU&(F~v/2Rq!"RBfx6fn7\{㼲FZ`,z۞=44.x]X4]K&ISWK%\5@XR,56ex.u\jpI@4ަcl:*ݹdwkr/pDΩۓDק٠!UJRY&{o;}uN_#/ǑX#}8Fbv~|"&8> _.>pY*+2R +_vhjGŊF$s5>l%<F.jfyL&Ve6ޅdR&Jkf|?F dj|ÏV@d }ǐ x._\]Vہ 4/ *ԏQ+jC (/ =PT@Ӎv?tX#?# A}~A12 v%<j@0uA/k!X @\^kkVDFT2 \JJp@A?C;.>:p^yP `;Q09ÏG45oPUp7Fe +endstream +endobj +614 0 obj +<< +/Length1 7204 +/Filter /FlateDecode +/Length 4875 +>> +stream +x9kXUוku}.y\^ jA#H-QV1iK'!CljhjR4k6~Ik-vZ˜~68:1t:c컟k^{`BKKVV>QD +,W@ˊ*.ܱ>꟣gYEY[>br~KcE fl-;=k6qWR۸=;o+|4 چΚ߅~`win*р1+_E]T,XƊ-b"{`*gEKsKs[e֏W--+u^L}j+)a,nfσ"eLEhMt6C@=v@4i)V![ Z^H\DA:$xW ֔z~!6-/3ECNtQ7 !jDo_(* [R8CuIy!RS0/)TaʢCKC-u?hOaaO5ϜW' |e32(9{$D@|&9e)\ᓋ ?߆OqyNc2>_b"?2T[yO>|vᑇh~؏3g@_}? Ypo> c7v +w>/.ˏ_$/Ӎ;i`gvȸC6o3at*tl!hK6 o7=bC7aC@ܶ RgL{Ua-UUXC+Dcd"c]X.U$= i!| K]x7e\u2x v5EVƅEVKի>\U+e\!>\n/L \څK +|I7.2B?.3/c</0<>̛kyGk1wbnsw3 +tJƴTO a!Љ)ja2S,2Qo8{ -&$ &x7OG/UOwcgE:,?zyи'c"<&1Pp;J:1<)Lka6 Sx[t;0tE;yfsZ j\-]hMDF`A=k5F6C PcDC(q( i3b`M%2B GX׎bY,bGe4V&؅ H9<[ bm1 , A%d +&%H>шTU!ׅ-51%8YKo0/˄~# .1=˃J| 8|;$h_7`ކq~? +Ǩįv*QҰΚaֳ*_Ȗ T#մf ]QI*9hpiW!+cDIG0v)\=X 3kIf4MqEԿV&M͔ hoYM(bx]Γ+C)EOM?Y<.\hL"bY1uH*_!\L@/C +X&ƂUY+&F3Zł12bpӹD+K + %H7÷(V(pQɒpס^K/ݏU:j$5z`bH[bHfcD-0wXhPpmr,$/bUڜG6bT7 ,Cdh5,z1 m\e c%$f9VϾ"Io'pD +.(tד%BK`):!lBNjӀjm wM-RSxkRV]R+4YIɑɱk<(1fh:SԜhk 9Y~W-!!ntBC].Chssr|zH}O$jݎ+ ;V +=wtWw,]_`MfNjN6,YeQ^);\U͐-d##t*73!L(o3AUyAɗI,/d$N'>k)1$ zO'AXL(6TPYwqneHA#9:91ʵAZH_F -;:#&Ó{lu젾6Пpp8\nWSH4k͑Y$9Y$a&gΜ,K8{8JΉ^ǵUVf"KjVn| ;~șes'~Hx aUNk]Fȕri5$K yfՁw_-<ўu_]2s٩y/W俑G +0'f:Nj>1T෹ QČ"l0\S0S@`2njBo_V};?#7lԴys.yDE%fn2d˱c6_]88 :[St+%%Gwctf\U M#x-eUN.rԈk.+â>cGRӐ;{6@BVޱk;'.U9!;{ܮcf&;.^re,?IRPSL9kYYNbGxʃe:^UNv?z|4DI<"uL0;4=+ =6 A1+K*k*6si{XuWB0)Fy0|Y,QON juZ.qzzz\@vwc.)r9(Rcf M#M+ɥ{\y9"sY*<Ϻ{ r%ZJke=خk :%YG0I5my2ҾHd^a#^foc'4sZ4l|5P( Qiࡈ0lHw~|6_oɷY/*Y1d(MRtOIє 棂q-XQl9 zDHGysB}n_/,.-/4,4.- Yh_X(cdf.-Φ#, +^ ~I~KaM-x}Y +w5)ߵk*S,o_j yڻn@L2\839cʞf|ETFD|eOޙJN~ve5o+wt3jP إ:'Y9[:WH:nou:&ګӋzCtȜ:#$y'E 1X2j\Y9c c L,3ʼˠ,ɺd}!٘lJ6}3ՕIOtn.FP!$*"ʪ5 |k+޼v#kOCvN"?:7-p1$f:LO7#3fz'Mbŭ!Zt>3W9c{AD‹lܽ|G/ʿG{ѩFzOGZs@6YQt%8nS$Ir [x]Kݩκ~2p"YD5%x%"gTJd)hFzD<㐉> ZFj( hK%.븠(׉$o+Ѷ.vˏ_gq],dMn%ɥē:=X$bd4Qd6h0& +ރ VɨIZh4ꆬ7+YQP=q`\Xǎ%).y=*'"2qQC> +&MXi3P/@ i#̦i-f֙ , ]lNhz:h'IC,jUN;w *^XMNE@C>&6WMu5AeaꡐZ[z0nQU1=V{J-SIt Ci +tU* +UDy1<0׫fuu?psifc?s 7u唄iSwY? 1ޚa b(J )1C<1Zw1,`#uMjzK +,!>DuI'TVTBƖȨm lo"l/3} +endstream +endobj +615 0 obj +<< +/Length1 16324 +/Filter /FlateDecode +/Length 11653 +>> +stream +x{y|TEo-w{;4!d# K a¾*/$@D0M%2bQ1ЀD@ 9:;uo||^ԽUw9uS|OB6 (`Ġ}~8շw|P^ 2ҁK= ڱ}??4cmhv)s>0!#Λ~^ـ"Θ7̹߂{gL^ɰ?@0ck.dDB9s @#\Ϛ ' ^bh9oSGe MousB~88oS ŏp?iQ-Y0 -9 ȯ/|bܓGa&Br$"1E7&Q]R~rd=XpNMEc9y:x=륝r jQ!ObX MWH?AӉ K:~ ާyQ,Z)ٙo+tRmwrsDA@i%[|-q [CPJ܂ r(k +܏,z[h{˃4N@iۧ'PF] ^ K%/G|6x6vD%tVCՠyP_z<57dw]_DPQ@+ +P4 +V޻ǭ Dаىݻ؋{SD:ZOa>EKfE݇3.woh)'Feq_;Z[}zN/*oi8ߧvO_}SgnlpnLM>HVcŞQ؁j$`;bP6dU[mIWFZP#z{UR V{D.hvhgw4֕]Fӟ+O*ğphVZْ MozSߡ?fFO7* ,63\m_zZx^+b#o:~9 +w_0^}?cӍzSd%86A)gG]ѳ(PW3 =IFevqF1GQ;=b o}ۇ۫{ķӷa=*Z_S|:8ݡo^Lӿ>}WѽFk}eI|%1їw[ŗV?/%?2.ӝ/Xĝ }^;}:yFZXNhճ*Fz0}viu<0 0}C+A=&&;ݨp=tgRFוXuX1?X? k]]*Ft1D3$K] .1%y/ tQa. +O08 ]0Ο^?kO2:'ftV:y8L3:bFNqS,$F'2:a^`Xh1z +]dcct+PF`Fdqi06-e h>ab?FBo1-Va'i|F{W'4V_OzzMbO3YB˛g&ǫgЉyWb݁wh7F怀sihV]b4s!v@3h:ܘh\N4j)a.Tt&w 4n;Zbڑ[%$=}|C;\Md4L=mgnFf1Ѹd1n%M1 zv1h6HJd8`40F@Y@Cbh>u-#-4 SGғ8juJC{~pPPx%^(t+h-1xz +W@ *G2 (P)l[Kh9gk*G-h@8@ J;jvP"A(gDBSI&PT/e7m7~ #x~ADp Ok : Fg&Ɏ*e7e$m[G&Zt hNF2L,dA; ̈́~B̓++hO&dZ@3h1Zؒō +k;ArHs퍲h*o$D&'%~0zC#s6H;CS"l{?19J.π,V+ǔugtoV۩^FOO}C>uRvƻ&){syE̳n}`m[a{3uN`r䎤SΙYbZcmIXi0D4->sU٪e#je!;).}9NQ"%Esn@D641yp{֍buB([LAAǘTbb;Q!nJAqa5dd@YYčk(~ڦ}<=_z-躲䅹Yƾ#:F-\Z9}576zq }^ s;뚳;4; % wQ~E]JPHCЕ q-.l0I;s&MBa Jorp&t'ζldڒ֭>vu+ |pc3tʜ + +lzoNnFa';˖ٙq.ٚYKYw\ݤvcE-F=)h5,*o-~QZx\:G{,2@^WK-Ǿpɟc.M'p}CVn~Z']ܓ/`o{ 9rnO)sWIՎW: jw%X\4>Ʈ盭hɡ7 %g*[vooxw޿;60sۅ$|o&;ٹjw/hEbc?cK1V}xLBDdrQi:䲥Lmv6k1')@*&J\~}U/g֮C_^vSWl|f=z,\xwX=laGᯣ>j|ݰUꈐtJR#8k6TXdΠh#+[Hk,̎<=vנ#O46(z0C\Cݽ2>`cz׃Ć%|23KȬi!MvYRmg{ҳ85$Kw=4Ľ[}}RLa[j@1B?/7T)>*ui2 ʸoK[GMO#'y,)maBP֡vސ e0!4$F#Z;ma((uƅGPzHOpeK..FN<,v>tј՝' }Rv5b/w裵O8ni>W }xW\fA,kÛws I#P*X4Ԁ5*TRCr{3Ù3UN9ٚåx +JUS.d3|2@b'NqN,ﮪ֠ 4>o)0e@KE$׵$ 㖣WS;͓w@Y JdrqBg"veID7%$9AɥWKGERnU` +}'ILGC|D: 獒n>}ebd+JU\Fߜtb' ;<\V$`X*(<ϔx>na.>~1t7CN dK`kN!:U0nȒmaSm0"efB۫KK~Sk뭟\_Å8}dk r22\W;jK4?agfQ" 1Ŋ\ {`F%g"pW^x~F6 wi1֒-|~{x_@!v@LUC*zKrP/ 5T'QZQ5ͮ YXh8k|-g +.:WmhAۃ;A+sҋ^ܼ4yǾ`78u߷&[oaݚxth &RtDV=11|Uua>+~,u;,o4#9W,\P'S@@2򚤀3Ʌx+Z"͂SE, Kzmu2^ڐuj .s]4ugsQ|f&ot4!a9vQOř s +kgϪ'tyûv5|\z?wvqsDئeB0τGHRCH(FD"X@xܔڠx8ϱ7ـsb8wg<0UtNPuTիz"ÜE&J Ȁ=ȐBQV1UI/K~aਹ[{_07 14FQRʗcQ'G %|Dl$<C&ɥBUugFx )Xi݈L#v σY8|;;,Mge[X-}xmptE''ĒN[ѡ[.ήǢ-\8lXށ`Q(f +Qmn|.l} Q #KO,Ijʗ!rwk4RiB"7ZUVnÆu%eW:>7)T_OR>y/.}N8ɇoxԂÏVPKU6 +",-mYՄ(x??؃G.2/gNRuy;=s]{p>lS&{RmjQզ\8@4#c;Uv\VRR^^RRFϑ^?7o> +wl٣ _oB?E Ǜ Vu:,U;! hJ,P F,A%rSВ7A8$ +F&yYDze%{-d !؀^ٷiحYC9ڛ•ȃ_S^_gџ_!uIoӛn-/M} u+uШ>INXle] /(\غ. 'x>.za?^| b#1"+V+\G-1M1 ^66MV,0D] +ktbӯ*VhJO׸4at5v›8٪Sp!;"8c`Ò1۲·( mpBje!IDQBIdbR@%dUQ{Rb{+*d9rҟʣ:K%͒ѧe29Wmsl_ϱ1$ ˃U)$Yj!I?P',i)" 3Y5v' _UOy̳xGobͬd)CE,m<nլ [G AAת݈{Gw j-ڋ7UHST9UIե8=~'"y)~V{2 vݓӿ0 Fos]j}nъZW5?,GaҤnyg+׳Wluf0iD{WOpZf,UP#ܛ=4gKf_aO?]7^knK>n/`g'm`#/ިqTcj~(e<SJR5jkmz=dk,űT3,V/^u(ZZmq¯,)poKN|~G¤ϟ\ߎeX$%^;ihW;P9>*G*BR@*>$R2ʒ=O@|(o?z o[\U-=xU!5Wf>nF,_wS rtݦ+wv23^FL1[VZt-]_ 3 +dȊ)u0J0hn֨}!g%.6a’r!UnʖgȤJ#O# *>f3?L}mVWz[Ep&L{{>'32e1p0 ri7 s?a! F{gWFʷ?F_0Q""\]|GA$Aޭiсz!l:|U`KhAp1JFc (w bV:rH#J7CPWoFGYt0K_JO)=O6/1jDMVgYLuYE Ujȭ-<;^uξ>I[5dzvՁ D2?@ +O0uNuTzdsdUY')T$SAr55,A#Odzx,xQjY#TD D&V6z}' +ZZay cs|3!zPi2r#.xi ~Z&i2-!FdUǭc!VWl''؁/?#Ֆ5XŦ,K-D*pC1`b-FU^(Ʉi2p}S;muޣ!@8;x `V: "L,]䍢|&(NQe:KHHr\hG(QGQHFg+KrE?1]ɭ+ZaYaU\\di `vh73K]c(P~WwQǠSv(ZCņwW͢wX; +~j[ml =F[p<ϯ̩cW~T<6v1U4<ŕOTpqKMiʑ& Rt7v~ɽRm{7;j0+zGYc) +Duw#Oox#f~= ("݂6t^hRSM$} {o9f#79ލsܣh"7={cQ喋?T=Qŭe47zQ,u#2X( }KhzB3LnB)pG,ʇ{EP@dl!wZO4X4>EZk3Ka_ w7fh 1 TfhwBӏ*atg}wvtkT8Ap|(P9pZL.2PCϷ=, ާh̯:'iwj/.Ԏ֎\7.Moo/e\w # @G43xwзw/H;p<#i&?1y +J쉹(uԹׇ~ mZD +endstream +endobj +616 0 obj +<< +/Type /FontDescriptor +/FontName /ICOIPK+DejaVuSans +/FontFamily (DejaVu Sans) +/Flags 4 +/FontBBox [ 0 -14 472 560 ] +/ItalicAngle 0 +/Ascent 928 +/Descent -235 +/CapHeight 560 +/StemV 80 +/StemH 80 +/FontFile2 612 0 R +>> +endobj +617 0 obj +<< +/Type /Font +/Subtype /CIDFontType2 +/BaseFont /ICOIPK+DejaVuSans +/CIDSystemInfo << +/Registry (Adobe) +/Ordering (Identity) +/Supplement 0 +>> +/W [ 3 [ 317 ] 17 [ 317 ] 20 [ 636 ] 25 [ 636 ] 37 [ 686 698 ] 42 [ 774 751 294 ] 54 [ 634 ] 68 [ 612 ] 70 [ 549 634 615 ] 74 [ 634 633 277 ] 78 [ 579 ] 80 [ 974 633 611 ] 85 [ 411 520 392 633 591 817 ] ] +/FontDescriptor 616 0 R +>> +endobj +618 0 obj +<< +/Length 702 +>> +stream +/CIDInit /ProcSet findresource begin +12 dict begin +begincmap +/CIDSystemInfo +<< /Registry (Adobe) +/Ordering (UCS) +/Supplement 0 +>> def +/CMapName /Adobe-Identity-UCS def +/CMapType 2 def +1 begincodespacerange +<0000> +endcodespacerange +27 beginbfchar +<002b> <0048> +<0044> <0061> +<0055> <0072> +<0047> <0064> +<0048> <0065> +<0051> <006e> +<004c> <0069> +<004a> <0067> +<0003> <0020> +<002a> <0047> +<0058> <0075> +<005a> <0077> +<0057> <0074> +<004b> <0068> +<0026> <0043> +<002c> <0049> +<0036> <0053> +<0059> <0076> +<0014> <0031> +<0011> <002e> +<0019> <0036> +<0025> <0042> +<0046> <0063> +<0050> <006d> +<004e> <006b> +<0052> <006f> +<0056> <0073> +endbfchar +endcmap +CMapName currentdict /CMap defineresource pop +end +end +endstream +endobj +619 0 obj +<< +/Type /Font +/Subtype /Type0 +/BaseFont /ICOIPK+DejaVuSans +/Encoding /Identity-H +/DescendantFonts [ 617 0 R ] +/ToUnicode 618 0 R +>> +endobj +620 0 obj +<< +/Type /FontDescriptor +/FontName /TGPMTG+Poppins +/FontFamily (Poppins) +/Flags 4 +/FontBBox [ -27 -260 122 795 ] +/ItalicAngle 0 +/Ascent 1049 +/Descent -349 +/CapHeight 795 +/StemV 80 +/StemH 80 +/FontFile2 613 0 R +>> +endobj +621 0 obj +<< +/Type /Font +/Subtype /CIDFontType2 +/BaseFont /TGPMTG+Poppins +/CIDSystemInfo << +/Registry (Adobe) +/Ordering (Identity) +/Supplement 0 +>> +/W [ 3 [ 266 ] 10 [ 158 453 453 ] 15 [ 197 550 209 ] 19 [ 627 319 574 588 628 627 634 545 630 629 212 ] 36 [ 673 612 771 706 512 503 777 691 245 ] 46 [ 598 431 ] 49 [ 702 785 578 ] 53 [ 607 586 540 674 675 975 ] 68 [ 675 675 606 675 619 328 675 639 245 247 514 245 1029 639 639 675 675 372 521 363 639 560 819 478 562 ] 206 [ 379 379 ] 209 [ 411 ] ] +/FontDescriptor 620 0 R +>> +endobj +622 0 obj +<< +/Length 1248 +>> +stream +/CIDInit /ProcSet findresource begin +12 dict begin +begincmap +/CIDSystemInfo +<< /Registry (Adobe) +/Ordering (UCS) +/Supplement 0 +>> def +/CMapName /Adobe-Identity-UCS def +/CMapType 2 def +1 begincodespacerange +<0000> +endcodespacerange +66 beginbfchar +<002b> <0048> +<0044> <0061> +<0055> <0072> +<0047> <0064> +<0048> <0065> +<0051> <006e> +<004c> <0069> +<004a> <0067> +<0003> <0020> +<002a> <0047> +<0058> <0075> +<005a> <0077> +<0057> <0074> +<004b> <0068> +<0026> <0043> +<002c> <0049> +<0036> <0053> +<0059> <0076> +<0014> <0031> +<0011> <002e> +<0019> <0036> +<0025> <0042> +<0046> <0063> +<0050> <006d> +<004e> <006b> +<0016> <0033> +<0017> <0034> +<0018> <0035> +<001a> <0037> +<0015> <0032> +<0013> <0030> +<0032> <004f> +<0052> <006f> +<0049> <0066> +<002e> <004b> +<004f> <006c> +<0035> <0052> +<0033> <0050> +<0056> <0073> +<0053> <0070> +<0024> <0041> +<0031> <004e> +<005c> <0079> +<0028> <0045> +<0037> <0054> +<0010> <002d> +<0045> <0062> +<0054> <0071> +<000b> <0028> +<000c> <0029> +<000f> <002c> +<001d> <003a> +<0039> <0056> +<001b> <0038> +<001c> <0039> +<0027> <0044> +<0029> <0046> +<00d1> <2022> +<005b> <0078> +<003a> <0057> +<0038> <0055> +<00ce> <201c> +<00cf> <201d> +<004d> <006a> +<002f> <004c> +<000a> <0027> +endbfchar +endcmap +CMapName currentdict /CMap defineresource pop +end +end +endstream +endobj +623 0 obj +<< +/Type /Font +/Subtype /Type0 +/BaseFont /TGPMTG+Poppins +/Encoding /Identity-H +/DescendantFonts [ 621 0 R ] +/ToUnicode 622 0 R +>> +endobj +624 0 obj +<< +/Type /FontDescriptor +/FontName /WKJFDI+DejaVuSansMono +/FontFamily (DejaVu Sans Mono) +/Flags 5 +/FontBBox [ 0 -14 584 759 ] +/ItalicAngle 0 +/Ascent 928 +/Descent -235 +/CapHeight 759 +/StemV 80 +/StemH 80 +/FontFile2 614 0 R +>> +endobj +625 0 obj +<< +/Type /Font +/Subtype /CIDFontType2 +/BaseFont /WKJFDI+DejaVuSansMono +/CIDSystemInfo << +/Registry (Adobe) +/Ordering (Identity) +/Supplement 0 +>> +/W [ 16 [ 601 601 ] 29 [ 601 ] 36 [ 601 ] 54 [ 601 601 ] 66 [ 601 ] 68 [ 601 ] 70 [ 601 601 601 601 ] 76 [ 601 ] 78 [ 601 601 601 601 601 601 ] 85 [ 601 601 601 601 601 ] 92 [ 601 ] ] +/FontDescriptor 624 0 R +>> +endobj +626 0 obj +<< +/Length 674 +>> +stream +/CIDInit /ProcSet findresource begin +12 dict begin +begincmap +/CIDSystemInfo +<< /Registry (Adobe) +/Ordering (UCS) +/Supplement 0 +>> def +/CMapName /Adobe-Identity-UCS def +/CMapType 2 def +1 begincodespacerange +<0000> +endcodespacerange +25 beginbfchar +<0048> <0065> +<0057> <0074> +<0046> <0063> +<0047> <0064> +<0049> <0066> +<0044> <0061> +<0058> <0075> +<004f> <006c> +<0056> <0073> +<0055> <0072> +<0011> <002e> +<005c> <0079> +<0050> <006d> +<0042> <005f> +<0053> <0070> +<0052> <006f> +<004c> <0069> +<001d> <003a> +<0010> <002d> +<0051> <006e> +<0036> <0053> +<0059> <0076> +<0024> <0041> +<0037> <0054> +<004e> <006b> +endbfchar +endcmap +CMapName currentdict /CMap defineresource pop +end +end +endstream +endobj +627 0 obj +<< +/Type /Font +/Subtype /Type0 +/BaseFont /WKJFDI+DejaVuSansMono +/Encoding /Identity-H +/DescendantFonts [ 625 0 R ] +/ToUnicode 626 0 R +>> +endobj +628 0 obj +<< +/Type /FontDescriptor +/FontName /TGTVMX+DejaVuSansMono +/FontFamily (DejaVu Sans Mono) +/Flags 4 +/FontBBox [ 0 -235 585 699 ] +/ItalicAngle 0 +/Ascent 928 +/Descent -235 +/CapHeight 699 +/StemV 80 +/StemH 80 +/FontFile2 615 0 R +>> +endobj +629 0 obj +<< +/Type /Font +/Subtype /CIDFontType2 +/BaseFont /TGTVMX+DejaVuSansMono +/CIDSystemInfo << +/Registry (Adobe) +/Ordering (Identity) +/Supplement 0 +>> +/W [ 3 [ 601 602 602 602 602 602 602 602 602 602 602 601 602 601 601 601 601 602 602 601 601 602 602 602 602 601 602 602 ] 32 [ 602 ] 36 [ 602 602 602 602 602 602 602 602 602 ] 46 [ 602 602 602 601 602 601 ] 53 [ 602 602 602 602 602 602 ] 60 [ 602 ] 62 [ 602 ] 64 [ 602 ] 66 [ 602 602 602 601 601 601 601 601 602 602 602 602 601 601 602 601 601 602 602 602 601 601 601 602 601 602 601 602 602 602 602 ] ] +/FontDescriptor 628 0 R +>> +endobj +630 0 obj +<< +/Length 1500 +>> +stream +/CIDInit /ProcSet findresource begin +12 dict begin +begincmap +/CIDSystemInfo +<< /Registry (Adobe) +/Ordering (UCS) +/Supplement 0 +>> def +/CMapName /Adobe-Identity-UCS def +/CMapType 2 def +1 begincodespacerange +<0000> +endcodespacerange +84 beginbfchar +<0056> <0073> +<005c> <0079> +<0046> <0063> +<0057> <0074> +<004f> <006c> +<0012> <002f> +<0048> <0065> +<0011> <002e> +<0047> <0064> +<001c> <0039> +<0013> <0030> +<0010> <002d> +<004e> <006b> +<0058> <0075> +<0045> <0062> +<0052> <006f> +<0051> <006e> +<0049> <0066> +<0059> <0076> +<0050> <006d> +<0055> <0072> +<004c> <0069> +<0042> <005f> +<0020> <003d> +<0014> <0031> +<0053> <0070> +<0044> <0061> +<005b> <0078> +<0015> <0032> +<0018> <0035> +<0003> <0020> +<0016> <0033> +<0017> <0034> +<004a> <0067> +<0005> <0022> +<004b> <0068> +<001d> <003a> +<0036> <0053> +<0024> <0041> +<0037> <0054> +<0039> <0056> +<000e> <002b> +<0006> <0023> +<0004> <0021> +<0007> <0024> +<000b> <0028> +<004d> <006a> +<005e> <007b> +<003e> <005b> +<000d> <002a> +<0040> <005d> +<000a> <0027> +<0060> <007d> +<000c> <0029> +<001e> <003b> +<005a> <0077> +<0031> <004e> +<0033> <0050> +<001b> <0038> +<002c> <0049> +<0028> <0045> +<002e> <004b> +<000f> <002c> +<0035> <0052> +<0054> <0071> +<0027> <0044> +<0026> <0043> +<002f> <004c> +<002b> <0048> +<003a> <0057> +<002a> <0047> +<0030> <004d> +<0019> <0036> +<0032> <004f> +<003c> <0059> +<001a> <0037> +<005f> <007c> +<0038> <0055> +<0009> <0026> +<0029> <0046> +<005d> <007a> +<0025> <0042> +<0043> <0060> +<0008> <0025> +endbfchar +endcmap +CMapName currentdict /CMap defineresource pop +end +end +endstream +endobj +631 0 obj +<< +/Type /Font +/Subtype /Type0 +/BaseFont /TGTVMX+DejaVuSansMono +/Encoding /Identity-H +/DescendantFonts [ 629 0 R ] +/ToUnicode 630 0 R +>> +endobj +632 0 obj +<< +/ICOIPK 619 0 R +/TGPMTG 623 0 R +/WKJFDI 627 0 R +/TGTVMX 631 0 R +>> +endobj +xref +0 633 +0000000000 65535 f +0000000015 00000 n +0000000235 00000 n +0000000351 00000 n +0000027525 00000 n +0000027658 00000 n +0000030309 00000 n +0000030508 00000 n +0000034100 00000 n +0000034450 00000 n +0000034584 00000 n +0000034720 00000 n +0000034882 00000 n +0000035045 00000 n +0000035201 00000 n +0000035358 00000 n +0000035515 00000 n +0000035675 00000 n +0000035836 00000 n +0000035997 00000 n +0000036148 00000 n +0000036300 00000 n +0000036475 00000 n +0000036651 00000 n +0000036827 00000 n +0000036999 00000 n +0000037172 00000 n +0000037344 00000 n +0000037517 00000 n +0000037690 00000 n +0000042711 00000 n +0000043015 00000 n +0000043265 00000 n +0000043401 00000 n +0000043564 00000 n +0000043721 00000 n +0000043878 00000 n +0000044039 00000 n +0000044200 00000 n +0000044352 00000 n +0000044525 00000 n +0000044698 00000 n +0000044871 00000 n +0000045133 00000 n +0000045393 00000 n +0000050574 00000 n +0000050829 00000 n +0000051084 00000 n +0000051217 00000 n +0000051348 00000 n +0000051479 00000 n +0000051610 00000 n +0000051741 00000 n +0000056745 00000 n +0000057042 00000 n +0000057175 00000 n +0000057306 00000 n +0000057439 00000 n +0000057570 00000 n +0000057701 00000 n +0000057832 00000 n +0000057965 00000 n +0000058098 00000 n +0000058229 00000 n +0000058360 00000 n +0000058491 00000 n +0000058622 00000 n +0000063591 00000 n +0000063860 00000 n +0000063993 00000 n +0000064124 00000 n +0000064255 00000 n +0000064386 00000 n +0000064517 00000 n +0000064713 00000 n +0000064972 00000 n +0000065203 00000 n +0000070823 00000 n +0000071176 00000 n +0000071309 00000 n +0000071440 00000 n +0000071571 00000 n +0000071702 00000 n +0000071833 00000 n +0000071964 00000 n +0000072095 00000 n +0000072226 00000 n +0000072357 00000 n +0000072489 00000 n +0000072621 00000 n +0000072753 00000 n +0000072885 00000 n +0000073017 00000 n +0000073150 00000 n +0000073281 00000 n +0000073412 00000 n +0000073543 00000 n +0000073674 00000 n +0000073903 00000 n +0000079315 00000 n +0000079752 00000 n +0000079990 00000 n +0000080124 00000 n +0000080256 00000 n +0000080388 00000 n +0000080520 00000 n +0000080652 00000 n +0000080784 00000 n +0000080916 00000 n +0000081048 00000 n +0000081180 00000 n +0000081313 00000 n +0000081446 00000 n +0000081579 00000 n +0000081712 00000 n +0000081845 00000 n +0000081978 00000 n +0000082111 00000 n +0000082244 00000 n +0000082377 00000 n +0000082510 00000 n +0000082643 00000 n +0000082776 00000 n +0000082909 00000 n +0000083042 00000 n +0000083175 00000 n +0000083308 00000 n +0000083441 00000 n +0000083573 00000 n +0000088687 00000 n +0000089166 00000 n +0000089299 00000 n +0000089432 00000 n +0000089565 00000 n +0000089698 00000 n +0000089831 00000 n +0000089964 00000 n +0000090097 00000 n +0000090230 00000 n +0000090363 00000 n +0000090496 00000 n +0000090629 00000 n +0000090762 00000 n +0000090895 00000 n +0000091028 00000 n +0000091161 00000 n +0000091294 00000 n +0000091427 00000 n +0000091560 00000 n +0000091693 00000 n +0000091826 00000 n +0000091959 00000 n +0000092092 00000 n +0000092225 00000 n +0000092358 00000 n +0000092491 00000 n +0000092624 00000 n +0000092757 00000 n +0000092890 00000 n +0000093023 00000 n +0000093156 00000 n +0000093289 00000 n +0000093421 00000 n +0000093552 00000 n +0000098511 00000 n +0000098950 00000 n +0000099083 00000 n +0000099216 00000 n +0000099349 00000 n +0000099482 00000 n +0000099615 00000 n +0000099748 00000 n +0000099881 00000 n +0000100014 00000 n +0000100147 00000 n +0000100280 00000 n +0000100413 00000 n +0000100546 00000 n +0000100679 00000 n +0000100812 00000 n +0000100945 00000 n +0000101078 00000 n +0000101211 00000 n +0000101344 00000 n +0000101477 00000 n +0000101610 00000 n +0000101743 00000 n +0000101876 00000 n +0000102009 00000 n +0000102142 00000 n +0000102275 00000 n +0000102408 00000 n +0000102540 00000 n +0000102671 00000 n +0000107769 00000 n +0000108272 00000 n +0000108405 00000 n +0000108538 00000 n +0000108671 00000 n +0000108804 00000 n +0000108937 00000 n +0000109070 00000 n +0000109203 00000 n +0000109336 00000 n +0000109469 00000 n +0000109602 00000 n +0000109735 00000 n +0000109869 00000 n +0000110003 00000 n +0000110137 00000 n +0000110271 00000 n +0000110405 00000 n +0000110539 00000 n +0000110673 00000 n +0000110807 00000 n +0000110941 00000 n +0000111075 00000 n +0000111209 00000 n +0000111343 00000 n +0000111477 00000 n +0000111611 00000 n +0000111745 00000 n +0000111879 00000 n +0000112013 00000 n +0000112147 00000 n +0000112281 00000 n +0000112415 00000 n +0000112549 00000 n +0000112683 00000 n +0000112817 00000 n +0000112950 00000 n +0000113082 00000 n +0000117606 00000 n +0000118109 00000 n +0000118243 00000 n +0000118377 00000 n +0000118511 00000 n +0000118645 00000 n +0000118779 00000 n +0000118913 00000 n +0000119047 00000 n +0000119181 00000 n +0000119315 00000 n +0000119449 00000 n +0000119583 00000 n +0000119717 00000 n +0000119851 00000 n +0000119985 00000 n +0000120119 00000 n +0000120253 00000 n +0000120387 00000 n +0000120521 00000 n +0000120655 00000 n +0000120789 00000 n +0000120923 00000 n +0000121057 00000 n +0000121191 00000 n +0000121325 00000 n +0000121459 00000 n +0000121593 00000 n +0000121727 00000 n +0000121861 00000 n +0000121995 00000 n +0000122129 00000 n +0000122263 00000 n +0000122397 00000 n +0000122531 00000 n +0000122665 00000 n +0000122798 00000 n +0000122930 00000 n +0000127159 00000 n +0000127678 00000 n +0000127812 00000 n +0000127946 00000 n +0000128080 00000 n +0000128214 00000 n +0000128348 00000 n +0000128482 00000 n +0000128616 00000 n +0000128750 00000 n +0000128884 00000 n +0000129018 00000 n +0000129152 00000 n +0000129286 00000 n +0000129420 00000 n +0000129554 00000 n +0000129688 00000 n +0000129822 00000 n +0000129956 00000 n +0000130090 00000 n +0000130224 00000 n +0000130358 00000 n +0000130492 00000 n +0000130626 00000 n +0000130760 00000 n +0000130894 00000 n +0000131028 00000 n +0000131162 00000 n +0000131296 00000 n +0000131430 00000 n +0000131564 00000 n +0000131698 00000 n +0000131832 00000 n +0000131966 00000 n +0000132100 00000 n +0000132234 00000 n +0000132368 00000 n +0000132502 00000 n +0000132635 00000 n +0000132767 00000 n +0000137769 00000 n +0000138288 00000 n +0000138422 00000 n +0000138556 00000 n +0000138690 00000 n +0000138824 00000 n +0000138958 00000 n +0000139092 00000 n +0000139226 00000 n +0000139360 00000 n +0000139494 00000 n +0000139628 00000 n +0000139762 00000 n +0000139896 00000 n +0000140030 00000 n +0000140164 00000 n +0000140298 00000 n +0000140432 00000 n +0000140566 00000 n +0000140700 00000 n +0000140834 00000 n +0000140968 00000 n +0000141102 00000 n +0000141236 00000 n +0000141370 00000 n +0000141504 00000 n +0000141638 00000 n +0000141772 00000 n +0000141906 00000 n +0000142040 00000 n +0000142174 00000 n +0000142308 00000 n +0000142442 00000 n +0000142576 00000 n +0000142710 00000 n +0000142844 00000 n +0000142978 00000 n +0000143112 00000 n +0000143245 00000 n +0000143377 00000 n +0000148406 00000 n +0000148925 00000 n +0000149059 00000 n +0000149193 00000 n +0000149327 00000 n +0000149461 00000 n +0000149595 00000 n +0000149729 00000 n +0000149863 00000 n +0000149997 00000 n +0000150131 00000 n +0000150265 00000 n +0000150399 00000 n +0000150533 00000 n +0000150667 00000 n +0000150801 00000 n +0000150935 00000 n +0000151069 00000 n +0000151203 00000 n +0000151337 00000 n +0000151471 00000 n +0000151605 00000 n +0000151739 00000 n +0000151873 00000 n +0000152007 00000 n +0000152141 00000 n +0000152275 00000 n +0000152409 00000 n +0000152543 00000 n +0000152677 00000 n +0000152811 00000 n +0000152945 00000 n +0000153079 00000 n +0000153213 00000 n +0000153347 00000 n +0000153481 00000 n +0000153615 00000 n +0000153749 00000 n +0000153882 00000 n +0000154014 00000 n +0000159201 00000 n +0000159680 00000 n +0000159814 00000 n +0000159948 00000 n +0000160082 00000 n +0000160216 00000 n +0000160350 00000 n +0000160484 00000 n +0000160618 00000 n +0000160752 00000 n +0000160886 00000 n +0000161020 00000 n +0000161154 00000 n +0000161288 00000 n +0000161422 00000 n +0000161556 00000 n +0000161786 00000 n +0000161921 00000 n +0000162054 00000 n +0000162187 00000 n +0000162320 00000 n +0000162453 00000 n +0000162586 00000 n +0000162719 00000 n +0000162852 00000 n +0000162985 00000 n +0000163119 00000 n +0000163253 00000 n +0000163387 00000 n +0000163521 00000 n +0000163655 00000 n +0000163789 00000 n +0000163923 00000 n +0000164056 00000 n +0000164188 00000 n +0000169046 00000 n +0000169557 00000 n +0000169691 00000 n +0000169825 00000 n +0000169959 00000 n +0000170093 00000 n +0000170227 00000 n +0000170361 00000 n +0000170495 00000 n +0000170629 00000 n +0000170763 00000 n +0000170897 00000 n +0000171031 00000 n +0000171165 00000 n +0000171299 00000 n +0000171433 00000 n +0000171567 00000 n +0000171701 00000 n +0000171835 00000 n +0000171969 00000 n +0000172103 00000 n +0000172237 00000 n +0000172371 00000 n +0000172505 00000 n +0000172639 00000 n +0000172773 00000 n +0000172907 00000 n +0000173041 00000 n +0000173175 00000 n +0000173309 00000 n +0000173443 00000 n +0000173577 00000 n +0000173711 00000 n +0000173845 00000 n +0000173979 00000 n +0000174113 00000 n +0000174247 00000 n +0000174380 00000 n +0000174512 00000 n +0000179215 00000 n +0000179694 00000 n +0000179828 00000 n +0000179962 00000 n +0000180096 00000 n +0000180230 00000 n +0000180364 00000 n +0000180498 00000 n +0000180632 00000 n +0000180766 00000 n +0000180900 00000 n +0000181034 00000 n +0000181168 00000 n +0000181302 00000 n +0000181436 00000 n +0000181570 00000 n +0000181704 00000 n +0000181838 00000 n +0000181972 00000 n +0000182106 00000 n +0000182240 00000 n +0000182374 00000 n +0000182508 00000 n +0000182642 00000 n +0000182776 00000 n +0000182910 00000 n +0000183044 00000 n +0000183178 00000 n +0000183312 00000 n +0000183446 00000 n +0000183580 00000 n +0000183714 00000 n +0000183848 00000 n +0000183981 00000 n +0000184113 00000 n +0000189306 00000 n +0000189785 00000 n +0000189919 00000 n +0000190053 00000 n +0000190187 00000 n +0000190321 00000 n +0000190455 00000 n +0000190589 00000 n +0000190723 00000 n +0000190857 00000 n +0000190991 00000 n +0000191125 00000 n +0000191259 00000 n +0000191394 00000 n +0000191529 00000 n +0000191664 00000 n +0000191799 00000 n +0000191934 00000 n +0000192069 00000 n +0000192204 00000 n +0000192339 00000 n +0000192474 00000 n +0000192609 00000 n +0000192744 00000 n +0000192879 00000 n +0000193014 00000 n +0000193149 00000 n +0000193284 00000 n +0000193419 00000 n +0000193554 00000 n +0000193689 00000 n +0000193824 00000 n +0000193959 00000 n +0000194093 00000 n +0000194226 00000 n +0000199366 00000 n +0000199749 00000 n +0000199884 00000 n +0000200019 00000 n +0000200154 00000 n +0000200289 00000 n +0000200424 00000 n +0000200559 00000 n +0000200694 00000 n +0000200829 00000 n +0000200964 00000 n +0000201097 00000 n +0000201230 00000 n +0000201363 00000 n +0000201496 00000 n +0000201629 00000 n +0000201762 00000 n +0000201895 00000 n +0000202028 00000 n +0000202162 00000 n +0000202296 00000 n +0000202430 00000 n +0000202563 00000 n +0000207539 00000 n +0000208010 00000 n +0000208144 00000 n +0000208278 00000 n +0000208412 00000 n +0000208546 00000 n +0000208680 00000 n +0000208814 00000 n +0000208948 00000 n +0000209082 00000 n +0000209216 00000 n +0000209350 00000 n +0000209484 00000 n +0000209618 00000 n +0000209752 00000 n +0000209887 00000 n +0000210020 00000 n +0000210153 00000 n +0000210286 00000 n +0000210419 00000 n +0000210552 00000 n +0000210685 00000 n +0000210818 00000 n +0000210951 00000 n +0000211085 00000 n +0000211219 00000 n +0000211353 00000 n +0000211487 00000 n +0000211621 00000 n +0000211755 00000 n +0000211889 00000 n +0000212023 00000 n +0000212157 00000 n +0000212290 00000 n +0000215522 00000 n +0000215769 00000 n +0000215903 00000 n +0000216037 00000 n +0000216171 00000 n +0000216305 00000 n +0000216476 00000 n +0000216630 00000 n +0000216745 00000 n +0000216911 00000 n +0000217100 00000 n +0000217229 00000 n +0000217422 00000 n +0000217595 00000 n +0000217779 00000 n +0000217938 00000 n +0000218117 00000 n +0000218294 00000 n +0000218486 00000 n +0000218704 00000 n +0000218916 00000 n +0000218978 00000 n +0000225626 00000 n +0000229868 00000 n +0000234832 00000 n +0000246576 00000 n +0000246808 00000 n +0000247198 00000 n +0000247953 00000 n +0000248103 00000 n +0000248332 00000 n +0000248864 00000 n +0000250166 00000 n +0000250313 00000 n +0000250554 00000 n +0000250927 00000 n +0000251654 00000 n +0000251808 00000 n +0000252050 00000 n +0000252644 00000 n +0000254198 00000 n +0000254352 00000 n +trailer +<< +/Size 633 +/Root 3 0 R +/Info 2 0 R +>> +startxref +254439 +%%EOF diff --git a/content/rancher/v2.6/en/security/hardening-guides/1.6-hardening-2.6/_index.md b/content/rancher/v2.6/en/security/hardening-guides/1.6-hardening-2.6/_index.md new file mode 100644 index 00000000000..51a6df997f8 --- /dev/null +++ b/content/rancher/v2.6/en/security/hardening-guides/1.6-hardening-2.6/_index.md @@ -0,0 +1,641 @@ +--- +title: Hardening Guide with CIS v1.6 Benchmark +weight: 100 +--- + +This document provides prescriptive guidance for hardening a production installation of a RKE cluster to be used with Rancher v2.6.3. It outlines the configurations and controls required to address Kubernetes benchmark controls from the Center for Information Security (CIS). + +> This hardening guide describes how to secure the nodes in your cluster, and it is recommended to follow this guide before installing Kubernetes. + +This hardening guide is intended to be used for RKE clusters and associated with specific versions of the CIS Kubernetes Benchmark, Kubernetes, and Rancher: + +| Rancher Version | CIS Benchmark Version | Kubernetes Version | +| --- | --- | --- | +| Rancher v2.6.3 | Benchmark v1.6 | Kubernetes v1.18, v1.19, v1.20 and v1.21 | + +[Click here to download a PDF version of this document](https://releases.rancher.com/documents/security/2.6/Rancher_v2-6_CIS_v1-6_Hardening_Guide.pdf). + +- [Overview](#overview) +- [Configure Kernel Runtime Parameters](#configure-kernel-runtime-parameters) +- [Configure `etcd` user and group](#configure-etcd-user-and-group) +- [Configure `default` service account](#configure-default-service-account) +- [Configure Network Policy](#configure-network-policy) +- [Reference Hardened RKE `cluster.yml` Configuration](#reference-hardened-rke-cluster-yml-configuration) +- [Reference Hardened RKE Template Configuration](#reference-hardened-rke-template-configuration) +- [Reference Hardened **cloud-config** Configuration](#reference-hardened-cloud-config-configuration) + +### Overview + +This document provides prescriptive guidance for hardening a RKE cluster to be used for installing Rancher v2.6.3 with Kubernetes v1.18 up to v1.21 or provisioning a RKE cluster with Kubernetes v1.18 up to v.21 to be used within Rancher v2.6.3. It outlines the configurations required to address Kubernetes benchmark controls from the Center for Information Security (CIS). + +For more details about evaluating a hardened cluster against the official CIS benchmark, refer to the [CIS 1.6 Benchmark - Self-Assessment Guide - Rancher v2.6]({{}}/rancher/v2.6/en/security/hardening-guides/1.6-benchmark-2.6/). + +#### Known Issues + +- Rancher **exec shell** and **view logs** for pods are **not** functional in a CIS v1.6 hardened setup when only public IP is provided when registering custom nodes. This functionality requires a private IP to be provided when registering the custom nodes. +- When setting the `default_pod_security_policy_template_id:` to `restricted` or `restricted-noroot`, based on the pod security policies (PSP) [provided]({{}}/rancher/v2.6/en/admin-settings/pod-security-policies/) by Rancher, Rancher creates **RoleBindings** and **ClusterRoleBindings** on the default service accounts. The CIS v1.6 check 5.1.5 requires that the default service accounts have no roles or cluster roles bound to it apart from the defaults. In addition the default service accounts should be configured such that it does not provide a service account token and does not have any explicit rights assignments. + +### Configure Kernel Runtime Parameters + +The following `sysctl` configuration is recommended for all nodes type in the cluster. Set the following parameters in `/etc/sysctl.d/90-kubelet.conf`: + +```ini +vm.overcommit_memory=1 +vm.panic_on_oom=0 +kernel.panic=10 +kernel.panic_on_oops=1 +kernel.keys.root_maxbytes=25000000 +``` + +Run `sysctl -p /etc/sysctl.d/90-kubelet.conf` to enable the settings. + +### Configure `etcd` user and group + +A user account and group for the **etcd** service is required to be setup before installing RKE. The **uid** and **gid** for the **etcd** user will be used in the RKE **config.yml** to set the proper permissions for files and directories during installation time. + +#### Create `etcd` user and group + +To create the **etcd** user and group run the following console commands. The commands below use `52034` for **uid** and **gid** are for example purposes. Any valid unused **uid** or **gid** could also be used in lieu of `52034`. + +```bash +groupadd --gid 52034 etcd +useradd --comment "etcd service account" --uid 52034 --gid 52034 etcd --shell /usr/sbin/nologin +``` + +Update the RKE **config.yml** with the **uid** and **gid** of the **etcd** user: + +```yaml +services: + etcd: + gid: 52034 + uid: 52034 +``` + +### Configure `default` Service Account + +#### Set `automountServiceAccountToken` to `false` for `default` service accounts + +Kubernetes provides a default service account which is used by cluster workloads where no specific service account is assigned to the pod. Where access to the Kubernetes API from a pod is required, a specific service account should be created for that pod, and rights granted to that service account. The default service account should be configured such that it does not provide a service account token and does not have any explicit rights assignments. + +For each namespace including **default** and **kube-system** on a standard RKE install, the **default** service account must include this value: + +```yaml +automountServiceAccountToken: false +``` + +Save the following configuration to a file called `account_update.yaml`. + +```yaml +apiVersion: v1 +kind: ServiceAccount +metadata: + name: default +automountServiceAccountToken: false +``` + +Create a bash script file called `account_update.sh`. Be sure to `chmod +x account_update.sh` so the script has execute permissions. + +```bash +#!/bin/bash -e + +for namespace in $(kubectl get namespaces -A -o=jsonpath="{.items[*]['metadata.name']}"); do + kubectl patch serviceaccount default -n ${namespace} -p "$(cat account_update.yaml)" +done +``` + +### Configure Network Policy + +#### Ensure that all Namespaces have Network Policies defined + +Running different applications on the same Kubernetes cluster creates a risk of one compromised application attacking a neighboring application. Network segmentation is important to ensure that containers can communicate only with those they are supposed to. A network policy is a specification of how selections of pods are allowed to communicate with each other and other network endpoints. + +Network Policies are namespace scoped. When a network policy is introduced to a given namespace, all traffic not allowed by the policy is denied. However, if there are no network policies in a namespace all traffic will be allowed into and out of the pods in that namespace. To enforce network policies, a CNI (container network interface) plugin must be enabled. This guide uses [Canal](https://github.com/projectcalico/canal) to provide the policy enforcement. Additional information about CNI providers can be found [here](https://www.suse.com/c/rancher_blog/comparing-kubernetes-cni-providers-flannel-calico-canal-and-weave/). + +Once a CNI provider is enabled on a cluster a default network policy can be applied. For reference purposes a **permissive** example is provided below. If you want to allow all traffic to all pods in a namespace (even if policies are added that cause some pods to be treated as “isolated”), you can create a policy that explicitly allows all traffic in that namespace. Save the following configuration as `default-allow-all.yaml`. Additional [documentation](https://kubernetes.io/docs/concepts/services-networking/network-policies/) about network policies can be found on the Kubernetes site. + +> This `NetworkPolicy` is just an example and is not recommended for production use. + +```yaml +--- +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: default-allow-all +spec: + podSelector: {} + ingress: + - {} + egress: + - {} + policyTypes: + - Ingress + - Egress +``` + +Create a bash script file called `apply_networkPolicy_to_all_ns.sh`. Be sure to `chmod +x apply_networkPolicy_to_all_ns.sh` so the script has execute permissions. + +```bash +#!/bin/bash -e + +for namespace in $(kubectl get namespaces -A -o=jsonpath="{.items[*]['metadata.name']}"); do + kubectl apply -f default-allow-all.yaml -n ${namespace} +done +``` + +Execute this script to apply the `default-allow-all.yaml` configuration with the **permissive** `NetworkPolicy` to all namespaces. + +### Reference Hardened RKE `cluster.yml` Configuration + +The reference `cluster.yml` is used by the RKE CLI that provides the configuration needed to achieve a hardened install of Rancher Kubernetes Engine (RKE). RKE install [documentation]({{}}/rke/latest/en/installation/) is provided with additional details about the configuration items. This reference `cluster.yml` does not include the required **nodes** directive which will vary depending on your environment. Documentation for node configuration in RKE can be found [here]({{}}/rke/latest/en/config-options/nodes/). + +> For a Kubernetes v1.18 cluster, the configuration `spec.volumes: 'ephemeral'` should be removed from the `PodSecurityPolicy`, since it's not supported in this Kubernetes release. + +```yaml +# If you intend to deploy Kubernetes in an air-gapped environment, +# please consult the documentation on how to configure custom RKE images. +# https://rancher.com/docs/rke/latest/en/installation/ . + +# The nodes directive is required and will vary depending on your environment. +# Documentation for node configuration can be found here: +# https://rancher.com/docs/rke/latest/en/config-options/nodes/ +nodes: [] +services: + etcd: + image: "" + extra_args: {} + extra_binds: [] + extra_env: [] + win_extra_args: {} + win_extra_binds: [] + win_extra_env: [] + external_urls: [] + ca_cert: "" + cert: "" + key: "" + path: "" + uid: 52034 + gid: 52034 + snapshot: false + retention: "" + creation: "" + backup_config: null + kube-api: + image: "" + extra_args: {} + extra_binds: [] + extra_env: [] + win_extra_args: {} + win_extra_binds: [] + win_extra_env: [] + service_cluster_ip_range: "" + service_node_port_range: "" + pod_security_policy: true + always_pull_images: false + secrets_encryption_config: + enabled: true + custom_config: null + audit_log: + enabled: true + configuration: null + admission_configuration: null + event_rate_limit: + enabled: true + configuration: null + kube-controller: + image: "" + extra_args: + feature-gates: RotateKubeletServerCertificate=true + tls-cipher-suites: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 + bind-address: 127.0.0.1 + extra_binds: [] + extra_env: [] + win_extra_args: {} + win_extra_binds: [] + win_extra_env: [] + cluster_cidr: "" + service_cluster_ip_range: "" + scheduler: + image: "" + extra_args: + tls-cipher-suites: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 + bind-address: 127.0.0.1 + extra_binds: [] + extra_env: [] + win_extra_args: {} + win_extra_binds: [] + win_extra_env: [] + kubelet: + image: "" + extra_args: + feature-gates: RotateKubeletServerCertificate=true + protect-kernel-defaults: true + tls-cipher-suites: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 + extra_binds: [] + extra_env: [] + win_extra_args: {} + win_extra_binds: [] + win_extra_env: [] + cluster_domain: cluster.local + infra_container_image: "" + cluster_dns_server: "" + fail_swap_on: false + generate_serving_certificate: true + kubeproxy: + image: "" + extra_args: {} + extra_binds: [] + extra_env: [] + win_extra_args: {} + win_extra_binds: [] + win_extra_env: [] +network: + plugin: "" + options: {} + mtu: 0 + node_selector: {} + update_strategy: null +authentication: + strategy: "" + sans: [] + webhook: null +addons: | + # Upstream Kubernetes restricted PSP policy + # https://github.com/kubernetes/website/blob/564baf15c102412522e9c8fc6ef2b5ff5b6e766c/content/en/examples/policy/restricted-psp.yaml + apiVersion: policy/v1beta1 + kind: PodSecurityPolicy + metadata: + name: restricted-noroot + spec: + privileged: false + # Required to prevent escalations to root. + allowPrivilegeEscalation: false + requiredDropCapabilities: + - ALL + # Allow core volume types. + volumes: + - 'configMap' + - 'emptyDir' + - 'projected' + - 'secret' + - 'downwardAPI' + # Assume that ephemeral CSI drivers & persistentVolumes set up by the cluster admin are safe to use. + - 'csi' + - 'persistentVolumeClaim' + - 'ephemeral' + hostNetwork: false + hostIPC: false + hostPID: false + runAsUser: + # Require the container to run without root privileges. + rule: 'MustRunAsNonRoot' + seLinux: + # This policy assumes the nodes are using AppArmor rather than SELinux. + rule: 'RunAsAny' + supplementalGroups: + rule: 'MustRunAs' + ranges: + # Forbid adding the root group. + - min: 1 + max: 65535 + fsGroup: + rule: 'MustRunAs' + ranges: + # Forbid adding the root group. + - min: 1 + max: 65535 + readOnlyRootFilesystem: false + --- + apiVersion: rbac.authorization.k8s.io/v1 + kind: ClusterRole + metadata: + name: psp:restricted-noroot + rules: + - apiGroups: + - extensions + resourceNames: + - restricted-noroot + resources: + - podsecuritypolicies + verbs: + - use + --- + apiVersion: rbac.authorization.k8s.io/v1 + kind: ClusterRoleBinding + metadata: + name: psp:restricted-noroot + roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: psp:restricted-noroot + subjects: + - apiGroup: rbac.authorization.k8s.io + kind: Group + name: system:serviceaccounts + - apiGroup: rbac.authorization.k8s.io + kind: Group + name: system:authenticated + --- + apiVersion: networking.k8s.io/v1 + kind: NetworkPolicy + metadata: + name: default-allow-all + spec: + podSelector: {} + ingress: + - {} + egress: + - {} + policyTypes: + - Ingress + - Egress + --- + apiVersion: v1 + kind: ServiceAccount + metadata: + name: default + automountServiceAccountToken: false +addons_include: [] +system_images: + etcd: "" + alpine: "" + nginx_proxy: "" + cert_downloader: "" + kubernetes_services_sidecar: "" + kubedns: "" + dnsmasq: "" + kubedns_sidecar: "" + kubedns_autoscaler: "" + coredns: "" + coredns_autoscaler: "" + nodelocal: "" + kubernetes: "" + flannel: "" + flannel_cni: "" + calico_node: "" + calico_cni: "" + calico_controllers: "" + calico_ctl: "" + calico_flexvol: "" + canal_node: "" + canal_cni: "" + canal_controllers: "" + canal_flannel: "" + canal_flexvol: "" + weave_node: "" + weave_cni: "" + pod_infra_container: "" + ingress: "" + ingress_backend: "" + metrics_server: "" + windows_pod_infra_container: "" +ssh_key_path: "" +ssh_cert_path: "" +ssh_agent_auth: false +authorization: + mode: "" + options: {} +ignore_docker_version: false +kubernetes_version: "" +private_registries: [] +ingress: + provider: "" + options: {} + node_selector: {} + extra_args: {} + dns_policy: "" + extra_envs: [] + extra_volumes: [] + extra_volume_mounts: [] + update_strategy: null + http_port: 0 + https_port: 0 + network_mode: "" +cluster_name: +cloud_provider: + name: "" +prefix_path: "" +win_prefix_path: "" +addon_job_timeout: 0 +bastion_host: + address: "" + port: "" + user: "" + ssh_key: "" + ssh_key_path: "" + ssh_cert: "" + ssh_cert_path: "" +monitoring: + provider: "" + options: {} + node_selector: {} + update_strategy: null + replicas: null +restore: + restore: false + snapshot_name: "" +dns: null +upgrade_strategy: + max_unavailable_worker: "" + max_unavailable_controlplane: "" + drain: null + node_drain_input: null +``` + +### Reference Hardened RKE Template Configuration + +The reference RKE template provides the configuration needed to achieve a hardened install of Kubernetes. RKE templates are used to provision Kubernetes and define Rancher settings. Follow the Rancher [documentation]({{}}/rancher/v2.6/en/installation) for additional installation and RKE template details. + +```yaml +# +# Cluster Config +# +default_pod_security_policy_template_id: restricted-noroot +docker_root_dir: /var/lib/docker +enable_cluster_alerting: false +enable_cluster_monitoring: false +enable_network_policy: true +local_cluster_auth_endpoint: + enabled: true +name: '' +# +# Rancher Config +# +rancher_kubernetes_engine_config: + addon_job_timeout: 45 + authentication: + strategy: x509 + dns: + nodelocal: + ip_address: '' + node_selector: null + update_strategy: {} + enable_cri_dockerd: false + ignore_docker_version: true +# +# # Currently only nginx ingress provider is supported. +# # To disable ingress controller, set `provider: none` +# # To enable ingress on specific nodes, use the node_selector, eg: +# provider: nginx +# node_selector: +# app: ingress +# + ingress: + default_backend: false + default_ingress_class: true + http_port: 0 + https_port: 0 + provider: nginx + kubernetes_version: v1.21.8-rancher1-1 + monitoring: + provider: metrics-server + replicas: 1 +# +# If you are using calico on AWS +# +# network: +# plugin: calico +# calico_network_provider: +# cloud_provider: aws +# +# # To specify flannel interface +# +# network: +# plugin: flannel +# flannel_network_provider: +# iface: eth1 +# +# # To specify flannel interface for canal plugin +# +# network: +# plugin: canal +# canal_network_provider: +# iface: eth1 +# + network: + mtu: 0 + options: + flannel_backend_type: vxlan + plugin: canal + rotate_encryption_key: false +# +# services: +# kube-api: +# service_cluster_ip_range: 10.43.0.0/16 +# kube-controller: +# cluster_cidr: 10.42.0.0/16 +# service_cluster_ip_range: 10.43.0.0/16 +# kubelet: +# cluster_domain: cluster.local +# cluster_dns_server: 10.43.0.10 +# + services: + scheduler: + extra_args: + tls-cipher-suites: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 + bind-address: 127.0.0.1 + etcd: + backup_config: + enabled: true + interval_hours: 12 + retention: 6 + safe_timestamp: false + timeout: 300 + creation: 12h + extra_args: + election-timeout: 5000 + heartbeat-interval: 500 + retention: 72h + snapshot: false + uid: 52034 + gid: 52034 + kube_api: + always_pull_images: false + audit_log: + enabled: true + event_rate_limit: + enabled: true + pod_security_policy: true + secrets_encryption_config: + enabled: true + service_node_port_range: 30000-32767 + kube-controller: + extra_args: + feature-gates: RotateKubeletServerCertificate=true + tls-cipher-suites: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 + bind-address: 127.0.0.1 + kubelet: + extra_args: + feature-gates: RotateKubeletServerCertificate=true + protect-kernel-defaults: true + tls-cipher-suites: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 + fail_swap_on: false + generate_serving_certificate: true + ssh_agent_auth: false + upgrade_strategy: + max_unavailable_controlplane: '1' + max_unavailable_worker: 10% +windows_prefered_cluster: false +``` + +### Reference Hardened **cloud-config** Configuration + +A **cloud-config** configuration file is generally used in cloud infrastructure environments to allow for configuration management of compute instances. The reference config configures SUSE Linux Enterprise Server (SLES), openSUSE Leap, Red Hat Enterprise Linux (RHEL) and Ubuntu operating system level settings needed before installing Kubernetes. + +#### Reference Hardened **cloud-config** for SUSE Linux Enterprise Server 15 (SLES 15) and openSUSE Leap 15 + +```yaml +#cloud-config +system_info: + default_user: + groups: + - docker +write_files: +- path: "/etc/sysctl.d/90-kubelet.conf" + owner: root:root + permissions: '0644' + content: | + vm.overcommit_memory=1 + vm.panic_on_oom=0 + kernel.panic=10 + kernel.panic_on_oops=1 + kernel.keys.root_maxbytes=25000000 +package_update: true +ssh_pwauth: false +runcmd: +# Docker should already be installed in SLES 15 SP3 +- zypper install docker containerd +- systemctl daemon-reload +- systemctl enable docker.service +- systemctl start --no-block docker.service +- sysctl -p /etc/sysctl.d/90-kubelet.conf +- groupadd --gid 52034 etcd +- useradd --comment "etcd service account" --uid 52034 --gid 52034 etcd --shell /usr/sbin/nologin +``` + +#### Reference Hardened **cloud-config** for Red Hat Enterprise Linux 8 (RHEL 8) and Ubuntu 20.04 LTS + +```yaml +#cloud-config +system_info: + default_user: + groups: + - docker +write_files: +- path: "/etc/sysctl.d/90-kubelet.conf" + owner: root:root + permissions: '0644' + content: | + vm.overcommit_memory=1 + vm.panic_on_oom=0 + kernel.panic=10 + kernel.panic_on_oops=1 + kernel.keys.root_maxbytes=25000000 +package_update: true +ssh_pwauth: false +runcmd: +# Install Docker from Rancher's Docker installation scripts - github.com/rancher/install-docker +- curl https://releases.rancher.com/install-docker/20.10.sh | sh +- sysctl -p /etc/sysctl.d/90-kubelet.conf +- groupadd --gid 52034 etcd +- useradd --comment "etcd service account" --uid 52034 --gid 52034 etcd --shell /usr/sbin/nologin +``` diff --git a/content/rancher/v2.6/en/security/hardening-guides/_index.md b/content/rancher/v2.6/en/security/hardening-guides/_index.md new file mode 100644 index 00000000000..ea4e969f058 --- /dev/null +++ b/content/rancher/v2.6/en/security/hardening-guides/_index.md @@ -0,0 +1,65 @@ +--- +title: Self-Assessment and Hardening Guides for Rancher v2.6 +shortTitle: Rancher v2.6 Hardening Guides +weight: 1 +aliases: + - /rancher/v2.6/en/security/rancher-2.5/ + - /rancher/v2.6/en/security/rancher-2.5/1.5-hardening-2.5/ + - /rancher/v2.6/en/security/rancher-2.5/1.5-benchmark-2.5/ + - /rancher/v2.6/en/security/rancher-2.5/1.6-hardening-2.5/ + - /rancher/v2.6/en/security/rancher-2.5/1.6-benchmark-2.5/ +--- + +Rancher provides specific security hardening guides for each supported Rancher's Kubernetes distributions. + +- [Rancher Kubernetes Distributions](#rancher-kubernetes-distributions) +- [Hardening Guides and Benchmark Versions](#hardening-guides-and-benchmark-versions) + - [RKE Guides](#rke-guides) + - [RKE2 Guides](#rke2-guides) + - [K3s Guides](#k3s) +- [Rancher with SELinux](#rancher-with-selinux) + +# Rancher Kubernetes Distributions + +Rancher uses the following Kubernetes distributions: + +- [**RKE**]({{}}/rke/latest/en/), Rancher Kubernetes Engine, is a CNCF-certified Kubernetes distribution that runs entirely within Docker containers. +- [**RKE2**](https://docs.rke2.io/) is a fully conformant Kubernetes distribution that focuses on security and compliance within the U.S. Federal Government sector. +- [**K3s**]({{}}/k3s/latest/en/) is a fully conformant, lightweight Kubernetes distribution. It is easy to install, with half the memory of upstream Kubernetes, all in a binary of less than 100 MB. + +To harden a Kubernetes cluster outside of Rancher's distributions, refer to your Kubernetes provider docs. + +# Hardening Guides and Benchmark Versions + +These guides have been tested along with the Rancher v2.6 release. Each self-assessment guide is accompanied with a hardening guide and tested on a specific Kubernetes version and CIS benchmark version. If a CIS benchmark has not been validated for your Kubernetes version, you can choose to use the existing guides until a newer version is added. + +### RKE Guides + +| Kubernetes Version | CIS Benchmark Version | Self Assessment Guide | Hardening Guides | +| --- | --- | --- | --- | +| Kubernetes v1.18, v1.19, v1.20 and v1.21 | CIS v1.6 | [Link](./1.6-benchmark-2.6) | [Link](./1.6-hardening-2.6) | + +> **Notes** +> +> - Kubernetes v1.22 is currently in experimental mode in Rancher v2.6.3. +> - CIS v1.20 benchmark version for Kubernetes v1.19 and v1.20 is not yet released as a profile in Rancher's CIS Benchmark chart. + +### RKE2 Guides + +| Kubernetes Version | CIS Benchmark Version | Self Assessment Guide | Hardening Guides | +| --- | --- | --- | --- | +| Kubernetes v1.18 | CIS v1.5 | [Link](https://docs.rke2.io/security/cis_self_assessment15/) | [Link](https://docs.rke2.io/security/hardening_guide/) | +| Kubernetes v1.20 | CIS v1.6 | [Link](https://docs.rke2.io/security/cis_self_assessment16/) | [Link](https://docs.rke2.io/security/hardening_guide/) | + +### K3s Guides + +| Kubernetes Version | CIS Benchmark Version | Self Assessment Guide | Hardening Guide | +| --- | --- | --- | --- | +| Kubernetes v1.17, v1.18, & v1.19 | CIS v1.5 | [Link]({{}}/k3s/latest/en/security/self_assessment/) | [Link]({{}}/k3s/latest/en/security/hardening_guide/) | + + +# Rancher with SELinux + +[Security-Enhanced Linux (SELinux)](https://en.wikipedia.org/wiki/Security-Enhanced_Linux) is a security enhancement to Linux. After being historically used by government agencies, SELinux is now industry standard and is enabled by default on RHEL and CentOS. + +To use Rancher with SELinux, we recommend installing the `rancher-selinux` RPM according to the instructions on [this page.]({{}}/rancher/v2.6/en/security/selinux/#installing-the-rancher-selinux-rpm) diff --git a/content/rancher/v2.6/en/security/rancher-2.5/1.5-benchmark-2.5/_index.md b/content/rancher/v2.6/en/security/rancher-2.5/1.5-benchmark-2.5/_index.md deleted file mode 100644 index 463446b78a0..00000000000 --- a/content/rancher/v2.6/en/security/rancher-2.5/1.5-benchmark-2.5/_index.md +++ /dev/null @@ -1,2265 +0,0 @@ ---- -title: CIS 1.5 Benchmark - Self-Assessment Guide - Rancher v2.5 -weight: 201 ---- - -### CIS v1.5 Kubernetes Benchmark - Rancher v2.5 with Kubernetes v1.15 - -[Click here to download a PDF version of this document](https://releases.rancher.com/documents/security/2.5/Rancher_1.5_Benchmark_Assessment.pdf) - -#### Overview - -This document is a companion to the Rancher v2.5 security hardening guide. The hardening guide provides prescriptive guidance for hardening a production installation of Rancher, and this benchmark guide is meant to help you evaluate the level of security of the hardened cluster against each control in the benchmark. - -This guide corresponds to specific versions of the hardening guide, Rancher, CIS Benchmark, and Kubernetes: - -Hardening Guide Version | Rancher Version | CIS Benchmark Version | Kubernetes Version ----------------------------|----------|---------|------- -Hardening Guide with CIS 1.5 Benchmark | Rancher v2.5 | CIS v1.5| Kubernetes v1.15 - -Because Rancher and RKE install Kubernetes services as Docker containers, many of the control verification checks in the CIS Kubernetes Benchmark don't apply and will have a result of `Not Applicable`. This guide will walk through the various controls and provide updated example commands to audit compliance in Rancher-created clusters. - -This document is to be used by Rancher operators, security teams, auditors and decision makers. - -For more detail about each audit, including rationales and remediations for failing tests, you can refer to the corresponding section of the CIS Kubernetes Benchmark v1.5. You can download the benchmark after logging in to [CISecurity.org]( https://www.cisecurity.org/benchmark/kubernetes/). - -#### Testing controls methodology - -Rancher and RKE install Kubernetes services via Docker containers. Configuration is defined by arguments passed to the container at the time of initialization, not via configuration files. - -Where control audits differ from the original CIS benchmark, the audit commands specific to Rancher Labs are provided for testing. -When performing the tests, you will need access to the Docker command line on the hosts of all three RKE roles. The commands also make use of the the [jq](https://stedolan.github.io/jq/) and [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/) (with valid config) tools to and are required in the testing and evaluation of test results. - -> NOTE: only scored tests are covered in this guide. - -### Controls - ---- -## 1 Master Node Security Configuration -### 1.1 Master Node Configuration Files - -#### 1.1.1 Ensure that the API server pod specification file permissions are set to `644` or more restrictive (Scored) - -**Result:** Not Applicable - -**Remediation:** -RKE doesn’t require or maintain a configuration file for the API server. All configuration is passed in as arguments at container run time. - -#### 1.1.2 Ensure that the API server pod specification file ownership is set to `root:root` (Scored) - -**Result:** Not Applicable - -**Remediation:** -RKE doesn’t require or maintain a configuration file for the API server. All configuration is passed in as arguments at container run time. - -#### 1.1.3 Ensure that the controller manager pod specification file permissions are set to `644` or more restrictive (Scored) - -**Result:** Not Applicable - -**Remediation:** -RKE doesn’t require or maintain a configuration file for the controller manager. All configuration is passed in as arguments at container run time. - -#### 1.1.4 Ensure that the controller manager pod specification file ownership is set to `root:root` (Scored) - -**Result:** Not Applicable - -**Remediation:** -RKE doesn’t require or maintain a configuration file for the controller manager. All configuration is passed in as arguments at container run time. - -#### 1.1.5 Ensure that the scheduler pod specification file permissions are set to `644` or more restrictive (Scored) - -**Result:** Not Applicable - -**Remediation:** -RKE doesn’t require or maintain a configuration file for the scheduler. All configuration is passed in as arguments at container run time. - -#### 1.1.6 Ensure that the scheduler pod specification file ownership is set to `root:root` (Scored) - -**Result:** Not Applicable - -**Remediation:** -RKE doesn’t require or maintain a configuration file for the scheduler. All configuration is passed in as arguments at container run time. - -#### 1.1.7 Ensure that the etcd pod specification file permissions are set to `644` or more restrictive (Scored) - -**Result:** Not Applicable - -**Remediation:** -RKE doesn’t require or maintain a configuration file for etcd. All configuration is passed in as arguments at container run time. - -#### 1.1.8 Ensure that the etcd pod specification file ownership is set to `root:root` (Scored) - -**Result:** Not Applicable - -**Remediation:** -RKE doesn’t require or maintain a configuration file for etcd. All configuration is passed in as arguments at container run time. - -#### 1.1.11 Ensure that the etcd data directory permissions are set to `700` or more restrictive (Scored) - -**Result:** PASS - -**Remediation:** -On the etcd server node, get the etcd data directory, passed as an argument `--data-dir`, -from the below command: - -``` bash -ps -ef | grep etcd -``` - -Run the below command (based on the etcd data directory found above). For example, - -``` bash -chmod 700 /var/lib/etcd -``` - -**Audit Script:** 1.1.11.sh - -``` -#!/bin/bash -e - -etcd_bin=${1} - -test_dir=$(ps -ef | grep ${etcd_bin} | grep -- --data-dir | sed 's%.*data-dir[= ]\([^ ]*\).*%\1%') - -docker inspect etcd | jq -r '.[].HostConfig.Binds[]' | grep "${test_dir}" | cut -d ":" -f 1 | xargs stat -c %a -``` - -**Audit Execution:** - -``` -./1.1.11.sh etcd -``` - -**Expected result**: - -``` -'700' is equal to '700' -``` - -#### 1.1.12 Ensure that the etcd data directory ownership is set to `etcd:etcd` (Scored) - -**Result:** PASS - -**Remediation:** -On the etcd server node, get the etcd data directory, passed as an argument `--data-dir`, -from the below command: - -``` bash -ps -ef | grep etcd -``` - -Run the below command (based on the etcd data directory found above). -For example, -``` bash -chown etcd:etcd /var/lib/etcd -``` - -**Audit Script:** 1.1.12.sh - -``` -#!/bin/bash -e - -etcd_bin=${1} - -test_dir=$(ps -ef | grep ${etcd_bin} | grep -- --data-dir | sed 's%.*data-dir[= ]\([^ ]*\).*%\1%') - -docker inspect etcd | jq -r '.[].HostConfig.Binds[]' | grep "${test_dir}" | cut -d ":" -f 1 | xargs stat -c %U:%G -``` - -**Audit Execution:** - -``` -./1.1.12.sh etcd -``` - -**Expected result**: - -``` -'etcd:etcd' is present -``` - -#### 1.1.13 Ensure that the `admin.conf` file permissions are set to `644` or more restrictive (Scored) - -**Result:** Not Applicable - -**Remediation:** -RKE does not store the kubernetes default kubeconfig credentials file on the nodes. It’s presented to user where RKE is run. -We recommend that this `kube_config_cluster.yml` file be kept in secure store. - -#### 1.1.14 Ensure that the admin.conf file ownership is set to `root:root` (Scored) - -**Result:** Not Applicable - -**Remediation:** -RKE does not store the kubernetes default kubeconfig credentials file on the nodes. It’s presented to user where RKE is run. -We recommend that this `kube_config_cluster.yml` file be kept in secure store. - -#### 1.1.15 Ensure that the `scheduler.conf` file permissions are set to `644` or more restrictive (Scored) - -**Result:** Not Applicable - -**Remediation:** -RKE doesn’t require or maintain a configuration file for the scheduler. All configuration is passed in as arguments at container run time. - -#### 1.1.16 Ensure that the `scheduler.conf` file ownership is set to `root:root` (Scored) - -**Result:** Not Applicable - -**Remediation:** -RKE doesn’t require or maintain a configuration file for the scheduler. All configuration is passed in as arguments at container run time. - -#### 1.1.17 Ensure that the `controller-manager.conf` file permissions are set to `644` or more restrictive (Scored) - -**Result:** Not Applicable - -**Remediation:** -RKE doesn’t require or maintain a configuration file for the controller manager. All configuration is passed in as arguments at container run time. - -#### 1.1.18 Ensure that the `controller-manager.conf` file ownership is set to `root:root` (Scored) - -**Result:** Not Applicable - -**Remediation:** -RKE doesn’t require or maintain a configuration file for the controller manager. All configuration is passed in as arguments at container run time. - -#### 1.1.19 Ensure that the Kubernetes PKI directory and file ownership is set to `root:root` (Scored) - -**Result:** PASS - -**Remediation:** -Run the below command (based on the file location on your system) on the master node. -For example, - -``` bash -chown -R root:root /etc/kubernetes/ssl -``` - -**Audit:** - -``` -stat -c %U:%G /etc/kubernetes/ssl -``` - -**Expected result**: - -``` -'root:root' is present -``` - -#### 1.1.20 Ensure that the Kubernetes PKI certificate file permissions are set to `644` or more restrictive (Scored) - -**Result:** PASS - -**Remediation:** -Run the below command (based on the file location on your system) on the master node. -For example, - -``` bash -chmod -R 644 /etc/kubernetes/ssl -``` - -**Audit Script:** check_files_permissions.sh - -``` -#!/usr/bin/env bash - -# This script is used to ensure the file permissions are set to 644 or -# more restrictive for all files in a given directory or a wildcard -# selection of files -# -# inputs: -# $1 = /full/path/to/directory or /path/to/fileswithpattern -# ex: !(*key).pem -# -# $2 (optional) = permission (ex: 600) -# -# outputs: -# true/false - -# Turn on "extended glob" for use of '!' in wildcard -shopt -s extglob - -# Turn off history to avoid surprises when using '!' -set -H - -USER_INPUT=$1 - -if [[ "${USER_INPUT}" == "" ]]; then - echo "false" - exit -fi - - -if [[ -d ${USER_INPUT} ]]; then - PATTERN="${USER_INPUT}/*" -else - PATTERN="${USER_INPUT}" -fi - -PERMISSION="" -if [[ "$2" != "" ]]; then - PERMISSION=$2 -fi - -FILES_PERMISSIONS=$(stat -c %n\ %a ${PATTERN}) - -while read -r fileInfo; do - p=$(echo ${fileInfo} | cut -d' ' -f2) - - if [[ "${PERMISSION}" != "" ]]; then - if [[ "$p" != "${PERMISSION}" ]]; then - echo "false" - exit - fi - else - if [[ "$p" != "644" && "$p" != "640" && "$p" != "600" ]]; then - echo "false" - exit - fi - fi -done <<< "${FILES_PERMISSIONS}" - - -echo "true" -exit -``` - -**Audit Execution:** - -``` -./check_files_permissions.sh '/etc/kubernetes/ssl/*.pem' -``` - -**Expected result**: - -``` -'true' is present -``` - -#### 1.1.21 Ensure that the Kubernetes PKI key file permissions are set to `600` (Scored) - -**Result:** PASS - -**Remediation:** -Run the below command (based on the file location on your system) on the master node. -For example, - -``` bash -chmod -R 600 /etc/kubernetes/ssl/certs/serverca -``` - -**Audit Script:** 1.1.21.sh - -``` -#!/bin/bash -e -check_dir=${1:-/etc/kubernetes/ssl} - -for file in $(find ${check_dir} -name "*key.pem"); do - file_permission=$(stat -c %a ${file}) - if [[ "${file_permission}" == "600" ]]; then - continue - else - echo "FAIL: ${file} ${file_permission}" - exit 1 - fi -done - -echo "pass" -``` - -**Audit Execution:** - -``` -./1.1.21.sh /etc/kubernetes/ssl -``` - -**Expected result**: - -``` -'pass' is present -``` - -### 1.2 API Server - -#### 1.2.2 Ensure that the `--basic-auth-file` argument is not set (Scored) - -**Result:** PASS - -**Remediation:** -Follow the documentation and configure alternate mechanisms for authentication. Then, -edit the API server pod specification file `/etc/kubernetes/manifests/kube-apiserver.yaml` -on the master node and remove the `--basic-auth-file=` parameter. - -**Audit:** - -``` -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected result**: - -``` -'--basic-auth-file' is not present -``` - -#### 1.2.3 Ensure that the `--token-auth-file` parameter is not set (Scored) - -**Result:** PASS - -**Remediation:** -Follow the documentation and configure alternate mechanisms for authentication. Then, -edit the API server pod specification file `/etc/kubernetes/manifests/kube-apiserver.yaml` -on the master node and remove the `--token-auth-file=` parameter. - -**Audit:** - -``` -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected result**: - -``` -'--token-auth-file' is not present -``` - -#### 1.2.4 Ensure that the `--kubelet-https` argument is set to true (Scored) - -**Result:** PASS - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the master node and remove the `--kubelet-https` parameter. - -**Audit:** - -``` -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected result**: - -``` -'--kubelet-https' is present OR '--kubelet-https' is not present -``` - -#### 1.2.5 Ensure that the `--kubelet-client-certificate` and `--kubelet-client-key` arguments are set as appropriate (Scored) - -**Result:** PASS - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection between the -apiserver and kubelets. Then, edit API server pod specification file -`/etc/kubernetes/manifests/kube-apiserver.yaml` on the master node and set the -kubelet client certificate and key parameters as below. - -``` bash ---kubelet-client-certificate= ---kubelet-client-key= -``` - -**Audit:** - -``` -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected result**: - -``` -'--kubelet-client-certificate' is present AND '--kubelet-client-key' is present -``` - -#### 1.2.6 Ensure that the `--kubelet-certificate-authority` argument is set as appropriate (Scored) - -**Result:** PASS - -**Remediation:** -Follow the Kubernetes documentation and setup the TLS connection between -the apiserver and kubelets. Then, edit the API server pod specification file -`/etc/kubernetes/manifests/kube-apiserver.yaml` on the master node and set the -`--kubelet-certificate-authority` parameter to the path to the cert file for the certificate authority. -`--kubelet-certificate-authority=` - -**Audit:** - -``` -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected result**: - -``` -'--kubelet-certificate-authority' is present -``` - -#### 1.2.7 Ensure that the `--authorization-mode` argument is not set to `AlwaysAllow` (Scored) - -**Result:** PASS - -**Remediation:** -Edit the API server pod specification file `/etc/kubernetes/manifests/kube-apiserver.yaml` -on the master node and set the `--authorization-mode` parameter to values other than `AlwaysAllow`. -One such example could be as below. - -``` bash ---authorization-mode=RBAC -``` - -**Audit:** - -``` -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected result**: - -``` -'Node,RBAC' not have 'AlwaysAllow' -``` - -#### 1.2.8 Ensure that the `--authorization-mode` argument includes `Node` (Scored) - -**Result:** PASS - -**Remediation:** -Edit the API server pod specification file `/etc/kubernetes/manifests/kube-apiserver.yaml` -on the master node and set the `--authorization-mode` parameter to a value that includes `Node`. - -``` bash ---authorization-mode=Node,RBAC -``` - -**Audit:** - -``` -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected result**: - -``` -'Node,RBAC' has 'Node' -``` - -#### 1.2.9 Ensure that the `--authorization-mode` argument includes `RBAC` (Scored) - -**Result:** PASS - -**Remediation:** -Edit the API server pod specification file `/etc/kubernetes/manifests/kube-apiserver.yaml` -on the master node and set the `--authorization-mode` parameter to a value that includes RBAC, -for example: - -``` bash ---authorization-mode=Node,RBAC -``` - -**Audit:** - -``` -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected result**: - -``` -'Node,RBAC' has 'RBAC' -``` - -#### 1.2.11 Ensure that the admission control plugin `AlwaysAdmit` is not set (Scored) - -**Result:** PASS - -**Remediation:** -Edit the API server pod specification file `/etc/kubernetes/manifests/kube-apiserver.yaml` -on the master node and either remove the `--enable-admission-plugins` parameter, or set it to a -value that does not include `AlwaysAdmit`. - -**Audit:** - -``` -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected result**: - -``` -'NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit' not have 'AlwaysAdmit' OR '--enable-admission-plugins' is not present -``` - -#### 1.2.14 Ensure that the admission control plugin `ServiceAccount` is set (Scored) - -**Result:** PASS - -**Remediation:** -Follow the documentation and create ServiceAccount objects as per your environment. -Then, edit the API server pod specification file `/etc/kubernetes/manifests/kube-apiserver.yaml` -on the master node and ensure that the `--disable-admission-plugins` parameter is set to a -value that does not include `ServiceAccount`. - -**Audit:** - -``` -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected result**: - -``` -'NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit' has 'ServiceAccount' OR '--enable-admission-plugins' is not present -``` - -#### 1.2.15 Ensure that the admission control plugin `NamespaceLifecycle` is set (Scored) - -**Result:** PASS - -**Remediation:** -Edit the API server pod specification file `/etc/kubernetes/manifests/kube-apiserver.yaml` -on the master node and set the `--disable-admission-plugins` parameter to -ensure it does not include `NamespaceLifecycle`. - -**Audit:** - -``` -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected result**: - -``` -'--disable-admission-plugins' is present OR '--disable-admission-plugins' is not present -``` - -#### 1.2.16 Ensure that the admission control plugin `PodSecurityPolicy` is set (Scored) - -**Result:** PASS - -**Remediation:** -Follow the documentation and create Pod Security Policy objects as per your environment. -Then, edit the API server pod specification file `/etc/kubernetes/manifests/kube-apiserver.yaml` -on the master node and set the `--enable-admission-plugins` parameter to a -value that includes `PodSecurityPolicy`: - -``` bash ---enable-admission-plugins=...,PodSecurityPolicy,... -``` - -Then restart the API Server. - -**Audit:** - -``` -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected result**: - -``` -'NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit' has 'PodSecurityPolicy' -``` - -#### 1.2.17 Ensure that the admission control plugin `NodeRestriction` is set (Scored) - -**Result:** PASS - -**Remediation:** -Follow the Kubernetes documentation and configure `NodeRestriction` plug-in on kubelets. -Then, edit the API server pod specification file `/etc/kubernetes/manifests/kube-apiserver.yaml` -on the master node and set the `--enable-admission-plugins` parameter to a -value that includes `NodeRestriction`. - -``` bash ---enable-admission-plugins=...,NodeRestriction,... -``` - -**Audit:** - -``` -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected result**: - -``` -'NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit' has 'NodeRestriction' -``` - -#### 1.2.18 Ensure that the `--insecure-bind-address` argument is not set (Scored) - -**Result:** PASS - -**Remediation:** -Edit the API server pod specification file `/etc/kubernetes/manifests/kube-apiserver.yaml` -on the master node and remove the `--insecure-bind-address` parameter. - -**Audit:** - -``` -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected result**: - -``` -'--insecure-bind-address' is not present -``` - -#### 1.2.19 Ensure that the `--insecure-port` argument is set to `0` (Scored) - -**Result:** PASS - -**Remediation:** -Edit the API server pod specification file `/etc/kubernetes/manifests/kube-apiserver.yaml` -on the master node and set the below parameter. - -``` bash ---insecure-port=0 -``` - -**Audit:** - -``` -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected result**: - -``` -'0' is equal to '0' -``` - -#### 1.2.20 Ensure that the `--secure-port` argument is not set to `0` (Scored) - -**Result:** PASS - -**Remediation:** -Edit the API server pod specification file `/etc/kubernetes/manifests/kube-apiserver.yaml` -on the master node and either remove the `--secure-port` parameter or -set it to a different **(non-zero)** desired port. - -**Audit:** - -``` -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected result**: - -``` -6443 is greater than 0 OR '--secure-port' is not present -``` - -#### 1.2.21 Ensure that the `--profiling` argument is set to `false` (Scored) - -**Result:** PASS - -**Remediation:** -Edit the API server pod specification file `/etc/kubernetes/manifests/kube-apiserver.yaml` -on the master node and set the below parameter. - -``` bash ---profiling=false -``` - -**Audit:** - -``` -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected result**: - -``` -'false' is equal to 'false' -``` - -#### 1.2.22 Ensure that the `--audit-log-path` argument is set (Scored) - -**Result:** PASS - -**Remediation:** -Edit the API server pod specification file `/etc/kubernetes/manifests/kube-apiserver.yaml` -on the master node and set the `--audit-log-path` parameter to a suitable path and -file where you would like audit logs to be written, for example: - -``` bash ---audit-log-path=/var/log/apiserver/audit.log -``` - -**Audit:** - -``` -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected result**: - -``` -'--audit-log-path' is present -``` - -#### 1.2.23 Ensure that the `--audit-log-maxage` argument is set to `30` or as appropriate (Scored) - -**Result:** PASS - -**Remediation:** -Edit the API server pod specification file `/etc/kubernetes/manifests/kube-apiserver.yaml` -on the master node and set the `--audit-log-maxage` parameter to `30` or as an appropriate number of days: - -``` bash ---audit-log-maxage=30 -``` - -**Audit:** - -``` -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected result**: - -``` -30 is greater or equal to 30 -``` - -#### 1.2.24 Ensure that the `--audit-log-maxbackup` argument is set to `10` or as appropriate (Scored) - -**Result:** PASS - -**Remediation:** -Edit the API server pod specification file `/etc/kubernetes/manifests/kube-apiserver.yaml` -on the master node and set the `--audit-log-maxbackup` parameter to `10` or to an appropriate -value. - -``` bash ---audit-log-maxbackup=10 -``` - -**Audit:** - -``` -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected result**: - -``` -10 is greater or equal to 10 -``` - -#### 1.2.25 Ensure that the `--audit-log-maxsize` argument is set to `100` or as appropriate (Scored) - -**Result:** PASS - -**Remediation:** -Edit the API server pod specification file `/etc/kubernetes/manifests/kube-apiserver.yaml` -on the master node and set the `--audit-log-maxsize` parameter to an appropriate size in **MB**. -For example, to set it as `100` **MB**: - -``` bash ---audit-log-maxsize=100 -``` - -**Audit:** - -``` -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected result**: - -``` -100 is greater or equal to 100 -``` - -#### 1.2.26 Ensure that the `--request-timeout` argument is set as appropriate (Scored) - -**Result:** PASS - -**Remediation:** -Edit the API server pod specification file `/etc/kubernetes/manifests/kube-apiserver.yaml` -and set the below parameter as appropriate and if needed. -For example, - -``` bash ---request-timeout=300s -``` - -**Audit:** - -``` -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected result**: - -``` -'--request-timeout' is not present OR '--request-timeout' is present -``` - -#### 1.2.27 Ensure that the `--service-account-lookup` argument is set to `true` (Scored) - -**Result:** PASS - -**Remediation:** -Edit the API server pod specification file `/etc/kubernetes/manifests/kube-apiserver.yaml` -on the master node and set the below parameter. - -``` bash ---service-account-lookup=true -``` - -Alternatively, you can delete the `--service-account-lookup` parameter from this file so -that the default takes effect. - -**Audit:** - -``` -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected result**: - -``` -'--service-account-lookup' is not present OR 'true' is equal to 'true' -``` - -#### 1.2.28 Ensure that the `--service-account-key-file` argument is set as appropriate (Scored) - -**Result:** PASS - -**Remediation:** -Edit the API server pod specification file `/etc/kubernetes/manifests/kube-apiserver.yaml` -on the master node and set the `--service-account-key-file` parameter -to the public key file for service accounts: - -``` bash ---service-account-key-file= -``` - -**Audit:** - -``` -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected result**: - -``` -'--service-account-key-file' is present -``` - -#### 1.2.29 Ensure that the `--etcd-certfile` and `--etcd-keyfile` arguments are set as appropriate (Scored) - -**Result:** PASS - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection between the apiserver and etcd. -Then, edit the API server pod specification file `/etc/kubernetes/manifests/kube-apiserver.yaml` -on the master node and set the **etcd** certificate and **key** file parameters. - -``` bash ---etcd-certfile= ---etcd-keyfile= -``` - -**Audit:** - -``` -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected result**: - -``` -'--etcd-certfile' is present AND '--etcd-keyfile' is present -``` - -#### 1.2.30 Ensure that the `--tls-cert-file` and `--tls-private-key-file` arguments are set as appropriate (Scored) - -**Result:** PASS - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection on the apiserver. -Then, edit the API server pod specification file `/etc/kubernetes/manifests/kube-apiserver.yaml` -on the master node and set the TLS certificate and private key file parameters. - -``` bash ---tls-cert-file= ---tls-private-key-file= -``` - -**Audit:** - -``` -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected result**: - -``` -'--tls-cert-file' is present AND '--tls-private-key-file' is present -``` - -#### 1.2.31 Ensure that the `--client-ca-file` argument is set as appropriate (Scored) - -**Result:** PASS - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection on the apiserver. -Then, edit the API server pod specification file `/etc/kubernetes/manifests/kube-apiserver.yaml` -on the master node and set the client certificate authority file. - -``` bash ---client-ca-file= -``` - -**Audit:** - -``` -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected result**: - -``` -'--client-ca-file' is present -``` - -#### 1.2.32 Ensure that the `--etcd-cafile` argument is set as appropriate (Scored) - -**Result:** PASS - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection between the apiserver and etcd. -Then, edit the API server pod specification file `/etc/kubernetes/manifests/kube-apiserver.yaml` -on the master node and set the etcd certificate authority file parameter. - -``` bash ---etcd-cafile= -``` - -**Audit:** - -``` -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected result**: - -``` -'--etcd-cafile' is present -``` - -#### 1.2.33 Ensure that the `--encryption-provider-config` argument is set as appropriate (Scored) - -**Result:** PASS - -**Remediation:** -Follow the Kubernetes documentation and configure a EncryptionConfig file. -Then, edit the API server pod specification file `/etc/kubernetes/manifests/kube-apiserver.yaml` -on the master node and set the `--encryption-provider-config` parameter to the path of that file: - -``` bash ---encryption-provider-config= -``` - -**Audit:** - -``` -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected result**: - -``` -'--encryption-provider-config' is present -``` - -#### 1.2.34 Ensure that encryption providers are appropriately configured (Scored) - -**Result:** PASS - -**Remediation:** -Follow the Kubernetes documentation and configure a `EncryptionConfig` file. -In this file, choose **aescbc**, **kms** or **secretbox** as the encryption provider. - -**Audit Script:** 1.2.34.sh - -``` -#!/bin/bash -e - -check_file=${1} - -grep -q -E 'aescbc|kms|secretbox' ${check_file} -if [ $? -eq 0 ]; then - echo "--pass" - exit 0 -else - echo "fail: encryption provider found in ${check_file}" - exit 1 -fi -``` - -**Audit Execution:** - -``` -./1.2.34.sh /etc/kubernetes/ssl/encryption.yaml -``` - -**Expected result**: - -``` -'--pass' is present -``` - -### 1.3 Controller Manager - -#### 1.3.1 Ensure that the `--terminated-pod-gc-threshold` argument is set as appropriate (Scored) - -**Result:** PASS - -**Remediation:** -Edit the Controller Manager pod specification file `/etc/kubernetes/manifests/kube-controller-manager.yaml` -on the master node and set the `--terminated-pod-gc-threshold` to an appropriate threshold, -for example: - -``` bash ---terminated-pod-gc-threshold=10 -``` - -**Audit:** - -``` -/bin/ps -ef | grep kube-controller-manager | grep -v grep -``` - -**Expected result**: - -``` -'--terminated-pod-gc-threshold' is present -``` - -#### 1.3.2 Ensure that the `--profiling` argument is set to false (Scored) - -**Result:** PASS - -**Remediation:** -Edit the Controller Manager pod specification file `/etc/kubernetes/manifests/kube-controller-manager.yaml` -on the master node and set the below parameter. - -``` bash ---profiling=false -``` - -**Audit:** - -``` -/bin/ps -ef | grep kube-controller-manager | grep -v grep -``` - -**Expected result**: - -``` -'false' is equal to 'false' -``` - -#### 1.3.3 Ensure that the `--use-service-account-credentials` argument is set to `true` (Scored) - -**Result:** PASS - -**Remediation:** -Edit the Controller Manager pod specification file `/etc/kubernetes/manifests/kube-controller-manager.yaml` -on the master node to set the below parameter. - -``` bash ---use-service-account-credentials=true -``` - -**Audit:** - -``` -/bin/ps -ef | grep kube-controller-manager | grep -v grep -``` - -**Expected result**: - -``` -'true' is not equal to 'false' -``` - -#### 1.3.4 Ensure that the `--service-account-private-key-file` argument is set as appropriate (Scored) - -**Result:** PASS - -**Remediation:** -Edit the Controller Manager pod specification file `/etc/kubernetes/manifests/kube-controller-manager.yaml` -on the master node and set the `--service-account-private-key-file` parameter -to the private key file for service accounts. - -``` bash ---service-account-private-key-file= -``` - -**Audit:** - -``` -/bin/ps -ef | grep kube-controller-manager | grep -v grep -``` - -**Expected result**: - -``` -'--service-account-private-key-file' is present -``` - -#### 1.3.5 Ensure that the `--root-ca-file` argument is set as appropriate (Scored) - -**Result:** PASS - -**Remediation:** -Edit the Controller Manager pod specification file `/etc/kubernetes/manifests/kube-controller-manager.yaml` -on the master node and set the `--root-ca-file` parameter to the certificate bundle file`. - -``` bash ---root-ca-file= -``` - -**Audit:** - -``` -/bin/ps -ef | grep kube-controller-manager | grep -v grep -``` - -**Expected result**: - -``` -'--root-ca-file' is present -``` - -#### 1.3.6 Ensure that the `RotateKubeletServerCertificate` argument is set to `true` (Scored) - -**Result:** PASS - -**Remediation:** -Edit the Controller Manager pod specification file `/etc/kubernetes/manifests/kube-controller-manager.yaml` -on the master node and set the `--feature-gates` parameter to include `RotateKubeletServerCertificate=true`. - -``` bash ---feature-gates=RotateKubeletServerCertificate=true -``` - -**Audit:** - -``` -/bin/ps -ef | grep kube-controller-manager | grep -v grep -``` - -**Expected result**: - -``` -'RotateKubeletServerCertificate=true' is equal to 'RotateKubeletServerCertificate=true' -``` - -#### 1.3.7 Ensure that the `--bind-address argument` is set to `127.0.0.1` (Scored) - -**Result:** PASS - -**Remediation:** -Edit the Controller Manager pod specification file `/etc/kubernetes/manifests/kube-controller-manager.yaml` -on the master node and ensure the correct value for the `--bind-address` parameter. - -**Audit:** - -``` -/bin/ps -ef | grep kube-controller-manager | grep -v grep -``` - -**Expected result**: - -``` -'--bind-address' is present OR '--bind-address' is not present -``` - -### 1.4 Scheduler - -#### 1.4.1 Ensure that the `--profiling` argument is set to `false` (Scored) - -**Result:** PASS - -**Remediation:** -Edit the Scheduler pod specification file `/etc/kubernetes/manifests/kube-scheduler.yaml` file -on the master node and set the below parameter. - -``` bash ---profiling=false -``` - -**Audit:** - -``` -/bin/ps -ef | grep kube-scheduler | grep -v grep -``` - -**Expected result**: - -``` -'false' is equal to 'false' -``` - -#### 1.4.2 Ensure that the `--bind-address` argument is set to `127.0.0.1` (Scored) - -**Result:** PASS - -**Remediation:** -Edit the Scheduler pod specification file `/etc/kubernetes/manifests/kube-scheduler.yaml` -on the master node and ensure the correct value for the `--bind-address` parameter. - -**Audit:** - -``` -/bin/ps -ef | grep kube-scheduler | grep -v grep -``` - -**Expected result**: - -``` -'--bind-address' is present OR '--bind-address' is not present -``` - -## 2 Etcd Node Configuration -### 2 Etcd Node Configuration Files - -#### 2.1 Ensure that the `--cert-file` and `--key-file` arguments are set as appropriate (Scored) - -**Result:** PASS - -**Remediation:** -Follow the etcd service documentation and configure TLS encryption. -Then, edit the etcd pod specification file `/etc/kubernetes/manifests/etcd.yaml` -on the master node and set the below parameters. - -``` bash ---cert-file= ---key-file= -``` - -**Audit:** - -``` -/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep -``` - -**Expected result**: - -``` -'--cert-file' is present AND '--key-file' is present -``` - -#### 2.2 Ensure that the `--client-cert-auth` argument is set to `true` (Scored) - -**Result:** PASS - -**Remediation:** -Edit the etcd pod specification file `/etc/kubernetes/manifests/etcd.yaml` on the master -node and set the below parameter. - -``` bash ---client-cert-auth="true" -``` - -**Audit:** - -``` -/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep -``` - -**Expected result**: - -``` -'true' is equal to 'true' -``` - -#### 2.3 Ensure that the `--auto-tls` argument is not set to `true` (Scored) - -**Result:** PASS - -**Remediation:** -Edit the etcd pod specification file `/etc/kubernetes/manifests/etcd.yaml` on the master -node and either remove the `--auto-tls` parameter or set it to `false`. - -``` bash - --auto-tls=false -``` - -**Audit:** - -``` -/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep -``` - -**Expected result**: - -``` -'--auto-tls' is not present OR '--auto-tls' is not present -``` - -#### 2.4 Ensure that the `--peer-cert-file` and `--peer-key-file` arguments are set as appropriate (Scored) - -**Result:** PASS - -**Remediation:** -Follow the etcd service documentation and configure peer TLS encryption as appropriate -for your etcd cluster. Then, edit the etcd pod specification file `/etc/kubernetes/manifests/etcd.yaml` on the -master node and set the below parameters. - -``` bash ---peer-client-file= ---peer-key-file= -``` - -**Audit:** - -``` -/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep -``` - -**Expected result**: - -``` -'--peer-cert-file' is present AND '--peer-key-file' is present -``` - -#### 2.5 Ensure that the `--peer-client-cert-auth` argument is set to `true` (Scored) - -**Result:** PASS - -**Remediation:** -Edit the etcd pod specification file `/etc/kubernetes/manifests/etcd.yaml` on the master -node and set the below parameter. - -``` bash ---peer-client-cert-auth=true -``` - -**Audit:** - -``` -/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep -``` - -**Expected result**: - -``` -'true' is equal to 'true' -``` - -#### 2.6 Ensure that the `--peer-auto-tls` argument is not set to `true` (Scored) - -**Result:** PASS - -**Remediation:** -Edit the etcd pod specification file `/etc/kubernetes/manifests/etcd.yaml` on the master -node and either remove the `--peer-auto-tls` parameter or set it to `false`. - -``` bash ---peer-auto-tls=false -``` - -**Audit:** - -``` -/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep -``` - -**Expected result**: - -``` -'--peer-auto-tls' is not present OR '--peer-auto-tls' is present -``` - -## 3 Control Plane Configuration -### 3.2 Logging - -#### 3.2.1 Ensure that a minimal audit policy is created (Scored) - -**Result:** PASS - -**Remediation:** -Create an audit policy file for your cluster. - -**Audit Script:** 3.2.1.sh - -``` -#!/bin/bash -e - -api_server_bin=${1} - -/bin/ps -ef | /bin/grep ${api_server_bin} | /bin/grep -v ${0} | /bin/grep -v grep -``` - -**Audit Execution:** - -``` -./3.2.1.sh kube-apiserver -``` - -**Expected result**: - -``` -'--audit-policy-file' is present -``` - -## 4 Worker Node Security Configuration -### 4.1 Worker Node Configuration Files - -#### 4.1.1 Ensure that the kubelet service file permissions are set to `644` or more restrictive (Scored) - -**Result:** Not Applicable - -**Remediation:** -RKE doesn’t require or maintain a configuration file for the kubelet service. All configuration is passed in as arguments at container run time. - -#### 4.1.2 Ensure that the kubelet service file ownership is set to `root:root` (Scored) - -**Result:** Not Applicable - -**Remediation:** -RKE doesn’t require or maintain a configuration file for the kubelet service. All configuration is passed in as arguments at container run time. - -#### 4.1.3 Ensure that the proxy kubeconfig file permissions are set to `644` or more restrictive (Scored) - -**Result:** PASS - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, - -``` bash -chmod 644 /etc/kubernetes/ssl/kubecfg-kube-proxy.yaml -``` - -**Audit:** - -``` -/bin/sh -c 'if test -e /etc/kubernetes/ssl/kubecfg-kube-proxy.yaml; then stat -c %a /etc/kubernetes/ssl/kubecfg-kube-proxy.yaml; fi' -``` - -**Expected result**: - -``` -'644' is present OR '640' is present OR '600' is equal to '600' OR '444' is present OR '440' is present OR '400' is present OR '000' is present -``` - -#### 4.1.4 Ensure that the proxy kubeconfig file ownership is set to `root:root` (Scored) - -**Result:** PASS - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, - -``` bash -chown root:root /etc/kubernetes/ssl/kubecfg-kube-proxy.yaml -``` - -**Audit:** - -``` -/bin/sh -c 'if test -e /etc/kubernetes/ssl/kubecfg-kube-proxy.yaml; then stat -c %U:%G /etc/kubernetes/ssl/kubecfg-kube-proxy.yaml; fi' -``` - -**Expected result**: - -``` -'root:root' is present -``` - -#### 4.1.5 Ensure that the kubelet.conf file permissions are set to `644` or more restrictive (Scored) - -**Result:** PASS - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, - -``` bash -chmod 644 /etc/kubernetes/ssl/kubecfg-kube-node.yaml -``` - -**Audit:** - -``` -/bin/sh -c 'if test -e /etc/kubernetes/ssl/kubecfg-kube-node.yaml; then stat -c %a /etc/kubernetes/ssl/kubecfg-kube-node.yaml; fi' -``` - -**Expected result**: - -``` -'644' is present OR '640' is present OR '600' is equal to '600' OR '444' is present OR '440' is present OR '400' is present OR '000' is present -``` - -#### 4.1.6 Ensure that the kubelet.conf file ownership is set to `root:root` (Scored) - -**Result:** PASS - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, - -``` bash -chown root:root /etc/kubernetes/ssl/kubecfg-kube-node.yaml -``` - -**Audit:** - -``` -/bin/sh -c 'if test -e /etc/kubernetes/ssl/kubecfg-kube-node.yaml; then stat -c %U:%G /etc/kubernetes/ssl/kubecfg-kube-node.yaml; fi' -``` - -**Expected result**: - -``` -'root:root' is equal to 'root:root' -``` - -#### 4.1.7 Ensure that the certificate authorities file permissions are set to `644` or more restrictive (Scored) - -**Result:** PASS - -**Remediation:** -Run the following command to modify the file permissions of the - -``` bash ---client-ca-file chmod 644 -``` - -**Audit:** - -``` -stat -c %a /etc/kubernetes/ssl/kube-ca.pem -``` - -**Expected result**: - -``` -'644' is equal to '644' OR '640' is present OR '600' is present -``` - -#### 4.1.8 Ensure that the client certificate authorities file ownership is set to `root:root` (Scored) - -**Result:** PASS - -**Remediation:** -Run the following command to modify the ownership of the `--client-ca-file`. - -``` bash -chown root:root -``` - -**Audit:** - -``` -/bin/sh -c 'if test -e /etc/kubernetes/ssl/kube-ca.pem; then stat -c %U:%G /etc/kubernetes/ssl/kube-ca.pem; fi' -``` - -**Expected result**: - -``` -'root:root' is equal to 'root:root' -``` - -#### 4.1.9 Ensure that the kubelet configuration file has permissions set to `644` or more restrictive (Scored) - -**Result:** Not Applicable - -**Remediation:** -RKE doesn’t require or maintain a configuration file for the kubelet service. All configuration is passed in as arguments at container run time. - -#### 4.1.10 Ensure that the kubelet configuration file ownership is set to `root:root` (Scored) - -**Result:** Not Applicable - -**Remediation:** -RKE doesn’t require or maintain a configuration file for the kubelet service. All configuration is passed in as arguments at container run time. - -### 4.2 Kubelet - -#### 4.2.1 Ensure that the `--anonymous-auth argument` is set to false (Scored) - -**Result:** PASS - -**Remediation:** -If using a Kubelet config file, edit the file to set authentication: `anonymous`: enabled to -`false`. -If using executable arguments, edit the kubelet service file -`/etc/systemd/system/kubelet.service.d/10-kubeadm.conf` on each worker node and -set the below parameter in `KUBELET_SYSTEM_PODS_ARGS` variable. - -``` bash ---anonymous-auth=false -``` - -Based on your system, restart the kubelet service. For example: - -``` bash -systemctl daemon-reload -systemctl restart kubelet.service -``` - -**Audit:** - -``` -/bin/ps -fC kubelet -``` - -**Audit Config:** - -``` -/bin/cat /var/lib/kubelet/config.yaml -``` - -**Expected result**: - -``` -'false' is equal to 'false' -``` - -#### 4.2.2 Ensure that the `--authorization-mode` argument is not set to `AlwaysAllow` (Scored) - -**Result:** PASS - -**Remediation:** -If using a Kubelet config file, edit the file to set authorization: `mode` to `Webhook`. If -using executable arguments, edit the kubelet service file -`/etc/systemd/system/kubelet.service.d/10-kubeadm.conf` on each worker node and -set the below parameter in `KUBELET_AUTHZ_ARGS` variable. - -``` bash ---authorization-mode=Webhook -``` - -Based on your system, restart the kubelet service. For example: - -``` bash -systemctl daemon-reload -systemctl restart kubelet.service -``` - -**Audit:** - -``` -/bin/ps -fC kubelet -``` - -**Audit Config:** - -``` -/bin/cat /var/lib/kubelet/config.yaml -``` - -**Expected result**: - -``` -'Webhook' not have 'AlwaysAllow' -``` - -#### 4.2.3 Ensure that the `--client-ca-file` argument is set as appropriate (Scored) - -**Result:** PASS - -**Remediation:** -If using a Kubelet config file, edit the file to set authentication: `x509`: `clientCAFile` to -the location of the client CA file. -If using command line arguments, edit the kubelet service file -`/etc/systemd/system/kubelet.service.d/10-kubeadm.conf` on each worker node and -set the below parameter in `KUBELET_AUTHZ_ARGS` variable. - -``` bash ---client-ca-file= -``` - -Based on your system, restart the kubelet service. For example: - -``` bash -systemctl daemon-reload -systemctl restart kubelet.service -``` - -**Audit:** - -``` -/bin/ps -fC kubelet -``` - -**Audit Config:** - -``` -/bin/cat /var/lib/kubelet/config.yaml -``` - -**Expected result**: - -``` -'--client-ca-file' is present -``` - -#### 4.2.4 Ensure that the `--read-only-port` argument is set to `0` (Scored) - -**Result:** PASS - -**Remediation:** -If using a Kubelet config file, edit the file to set `readOnlyPort` to `0`. -If using command line arguments, edit the kubelet service file -`/etc/systemd/system/kubelet.service.d/10-kubeadm.conf` on each worker node and -set the below parameter in `KUBELET_SYSTEM_PODS_ARGS` variable. - -``` bash ---read-only-port=0 -``` - -Based on your system, restart the kubelet service. For example: - -``` bash -systemctl daemon-reload -systemctl restart kubelet.service -``` - -**Audit:** - -``` -/bin/ps -fC kubelet -``` - -**Audit Config:** - -``` -/bin/cat /var/lib/kubelet/config.yaml -``` - -**Expected result**: - -``` -'0' is equal to '0' -``` - -#### 4.2.5 Ensure that the `--streaming-connection-idle-timeout` argument is not set to `0` (Scored) - -**Result:** PASS - -**Remediation:** -If using a Kubelet config file, edit the file to set `streamingConnectionIdleTimeout` to a -value other than `0`. -If using command line arguments, edit the kubelet service file -`/etc/systemd/system/kubelet.service.d/10-kubeadm.conf` on each worker node and -set the below parameter in `KUBELET_SYSTEM_PODS_ARGS` variable. - -``` bash ---streaming-connection-idle-timeout=5m -``` - -Based on your system, restart the kubelet service. For example: - -``` bash -systemctl daemon-reload -systemctl restart kubelet.service -``` - -**Audit:** - -``` -/bin/ps -fC kubelet -``` - -**Audit Config:** - -``` -/bin/cat /var/lib/kubelet/config.yaml -``` - -**Expected result**: - -``` -'30m' is not equal to '0' OR '--streaming-connection-idle-timeout' is not present -``` - -#### 4.2.6 Ensure that the ```--protect-kernel-defaults``` argument is set to `true` (Scored) - -**Result:** PASS - -**Remediation:** -If using a Kubelet config file, edit the file to set `protectKernelDefaults`: `true`. -If using command line arguments, edit the kubelet service file -`/etc/systemd/system/kubelet.service.d/10-kubeadm.conf` on each worker node and -set the below parameter in `KUBELET_SYSTEM_PODS_ARGS` variable. - -``` bash ---protect-kernel-defaults=true -``` - -Based on your system, restart the kubelet service. For example: - -``` bash -systemctl daemon-reload -systemctl restart kubelet.service -``` - -**Audit:** - -``` -/bin/ps -fC kubelet -``` - -**Audit Config:** - -``` -/bin/cat /var/lib/kubelet/config.yaml -``` - -**Expected result**: - -``` -'true' is equal to 'true' -``` - -#### 4.2.7 Ensure that the `--make-iptables-util-chains` argument is set to `true` (Scored) - -**Result:** PASS - -**Remediation:** -If using a Kubelet config file, edit the file to set `makeIPTablesUtilChains`: `true`. -If using command line arguments, edit the kubelet service file -`/etc/systemd/system/kubelet.service.d/10-kubeadm.conf` on each worker node and -remove the `--make-iptables-util-chains` argument from the -`KUBELET_SYSTEM_PODS_ARGS` variable. -Based on your system, restart the kubelet service. For example: - -```bash -systemctl daemon-reload -systemctl restart kubelet.service -``` - -**Audit:** - -``` -/bin/ps -fC kubelet -``` - -**Audit Config:** - -``` -/bin/cat /var/lib/kubelet/config.yaml -``` - -**Expected result**: - -``` -'true' is equal to 'true' OR '--make-iptables-util-chains' is not present -``` - -#### 4.2.10 Ensure that the `--tls-cert-file` and `--tls-private-key-file` arguments are set as appropriate (Scored) - -**Result:** Not Applicable - -**Remediation:** -RKE doesn’t require or maintain a configuration file for the kubelet service. All configuration is passed in as arguments at container run time. - -#### 4.2.11 Ensure that the `--rotate-certificates` argument is not set to `false` (Scored) - -**Result:** PASS - -**Remediation:** -If using a Kubelet config file, edit the file to add the line `rotateCertificates`: `true` or -remove it altogether to use the default value. -If using command line arguments, edit the kubelet service file -`/etc/systemd/system/kubelet.service.d/10-kubeadm.conf` on each worker node and -remove `--rotate-certificates=false` argument from the `KUBELET_CERTIFICATE_ARGS` -variable. -Based on your system, restart the kubelet service. For example: - -``` bash -systemctl daemon-reload -systemctl restart kubelet.service -``` - -**Audit:** - -``` -/bin/ps -fC kubelet -``` - -**Audit Config:** - -``` -/bin/cat /var/lib/kubelet/config.yaml -``` - -**Expected result**: - -``` -'--rotate-certificates' is present OR '--rotate-certificates' is not present -``` - -#### 4.2.12 Ensure that the `RotateKubeletServerCertificate` argument is set to `true` (Scored) - -**Result:** PASS - -**Remediation:** -Edit the kubelet service file `/etc/systemd/system/kubelet.service.d/10-kubeadm.conf` -on each worker node and set the below parameter in `KUBELET_CERTIFICATE_ARGS` variable. - -``` bash ---feature-gates=RotateKubeletServerCertificate=true -``` - -Based on your system, restart the kubelet service. For example: - -``` bash -systemctl daemon-reload -systemctl restart kubelet.service -``` - -**Audit:** - -``` -/bin/ps -fC kubelet -``` - -**Audit Config:** - -``` -/bin/cat /var/lib/kubelet/config.yaml -``` - -**Expected result**: - -``` -'true' is equal to 'true' -``` - -## 5 Kubernetes Policies -### 5.1 RBAC and Service Accounts - -#### 5.1.5 Ensure that default service accounts are not actively used. (Scored) - -**Result:** PASS - -**Remediation:** -Create explicit service accounts wherever a Kubernetes workload requires specific access -to the Kubernetes API server. -Modify the configuration of each default service account to include this value - -``` bash -automountServiceAccountToken: false -``` - -**Audit Script:** 5.1.5.sh - -``` -#!/bin/bash - -export KUBECONFIG=${KUBECONFIG:-/root/.kube/config} - -kubectl version > /dev/null -if [ $? -ne 0 ]; then - echo "fail: kubectl failed" - exit 1 -fi - -accounts="$(kubectl --kubeconfig=${KUBECONFIG} get serviceaccounts -A -o json | jq -r '.items[] | select(.metadata.name=="default") | select((.automountServiceAccountToken == null) or (.automountServiceAccountToken == true)) | "fail \(.metadata.name) \(.metadata.namespace)"')" - -if [[ "${accounts}" != "" ]]; then - echo "fail: automountServiceAccountToken not false for accounts: ${accounts}" - exit 1 -fi - -default_binding="$(kubectl get rolebindings,clusterrolebindings -A -o json | jq -r '.items[] | select(.subjects[].kind=="ServiceAccount" and .subjects[].name=="default" and .metadata.name=="default").metadata.uid' | wc -l)" - -if [[ "${default_binding}" -gt 0 ]]; then - echo "fail: default service accounts have non default bindings" - exit 1 -fi - -echo "--pass" -exit 0 -``` - -**Audit Execution:** - -``` -./5.1.5.sh -``` - -**Expected result**: - -``` -'--pass' is present -``` - -### 5.2 Pod Security Policies - -#### 5.2.2 Minimize the admission of containers wishing to share the host process ID namespace (Scored) - -**Result:** PASS - -**Remediation:** -Create a PSP as described in the Kubernetes documentation, ensuring that the -`.spec.hostPID` field is omitted or set to `false`. - -**Audit:** - -``` -kubectl --kubeconfig=/root/.kube/config get psp -o json | jq .items[] | jq -r 'select((.spec.hostPID == null) or (.spec.hostPID == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' -``` - -**Expected result**: - -``` -1 is greater than 0 -``` - -#### 5.2.3 Minimize the admission of containers wishing to share the host IPC namespace (Scored) - -**Result:** PASS - -**Remediation:** -Create a PSP as described in the Kubernetes documentation, ensuring that the -`.spec.hostIPC` field is omitted or set to `false`. - -**Audit:** - -``` -kubectl --kubeconfig=/root/.kube/config get psp -o json | jq .items[] | jq -r 'select((.spec.hostIPC == null) or (.spec.hostIPC == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' -``` - -**Expected result**: - -``` -1 is greater than 0 -``` - -#### 5.2.4 Minimize the admission of containers wishing to share the host network namespace (Scored) - -**Result:** PASS - -**Remediation:** -Create a PSP as described in the Kubernetes documentation, ensuring that the -`.spec.hostNetwork` field is omitted or set to `false`. - -**Audit:** - -``` -kubectl --kubeconfig=/root/.kube/config get psp -o json | jq .items[] | jq -r 'select((.spec.hostNetwork == null) or (.spec.hostNetwork == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' -``` - -**Expected result**: - -``` -1 is greater than 0 -``` - -#### 5.2.5 Minimize the admission of containers with `allowPrivilegeEscalation` (Scored) - -**Result:** PASS - -**Remediation:** -Create a PSP as described in the Kubernetes documentation, ensuring that the -`.spec.allowPrivilegeEscalation` field is omitted or set to `false`. - -**Audit:** - -``` -kubectl --kubeconfig=/root/.kube/config get psp -o json | jq .items[] | jq -r 'select((.spec.allowPrivilegeEscalation == null) or (.spec.allowPrivilegeEscalation == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' -``` - -**Expected result**: - -``` -1 is greater than 0 -``` - -### 5.3 Network Policies and CNI - -#### 5.3.2 Ensure that all Namespaces have Network Policies defined (Scored) - -**Result:** PASS - -**Remediation:** -Follow the documentation and create `NetworkPolicy` objects as you need them. - -**Audit Script:** 5.3.2.sh - -``` -#!/bin/bash -e - -export KUBECONFIG=${KUBECONFIG:-"/root/.kube/config"} - -kubectl version > /dev/null -if [ $? -ne 0 ]; then - echo "fail: kubectl failed" - exit 1 -fi - -for namespace in $(kubectl get namespaces -A -o json | jq -r '.items[].metadata.name'); do - policy_count=$(kubectl get networkpolicy -n ${namespace} -o json | jq '.items | length') - if [ ${policy_count} -eq 0 ]; then - echo "fail: ${namespace}" - exit 1 - fi -done - -echo "pass" -``` - -**Audit Execution:** - -``` -./5.3.2.sh -``` - -**Expected result**: - -``` -'pass' is present -``` - -### 5.6 General Policies - -#### 5.6.4 The default namespace should not be used (Scored) - -**Result:** PASS - -**Remediation:** -Ensure that namespaces are created to allow for appropriate segregation of Kubernetes -resources and that all new resources are created in a specific namespace. - -**Audit Script:** 5.6.4.sh - -``` -#!/bin/bash -e - -export KUBECONFIG=${KUBECONFIG:-/root/.kube/config} - -kubectl version > /dev/null -if [[ $? -gt 0 ]]; then - echo "fail: kubectl failed" - exit 1 -fi - -default_resources=$(kubectl get all -o json | jq --compact-output '.items[] | select((.kind == "Service") and (.metadata.name == "kubernetes") and (.metadata.namespace == "default") | not)' | wc -l) - -echo "--count=${default_resources}" -``` - -**Audit Execution:** - -``` -./5.6.4.sh -``` - -**Expected result**: - -``` -'0' is equal to '0' -``` diff --git a/content/rancher/v2.6/en/security/rancher-2.5/1.5-hardening-2.5/_index.md b/content/rancher/v2.6/en/security/rancher-2.5/1.5-hardening-2.5/_index.md deleted file mode 100644 index 7f64bbbc6a9..00000000000 --- a/content/rancher/v2.6/en/security/rancher-2.5/1.5-hardening-2.5/_index.md +++ /dev/null @@ -1,720 +0,0 @@ ---- -title: Hardening Guide with CIS 1.5 Benchmark -weight: 200 ---- - -This document provides prescriptive guidance for hardening a production installation of a RKE cluster to be used with Rancher v2.5. It outlines the configurations and controls required to address Kubernetes benchmark controls from the Center for Information Security (CIS). - -> This hardening guide describes how to secure the nodes in your cluster, and it is recommended to follow this guide before installing Kubernetes. - -This hardening guide is intended to be used for RKE clusters and associated with specific versions of the CIS Kubernetes Benchmark, Kubernetes, and Rancher: - - Rancher Version | CIS Benchmark Version | Kubernetes Version -----------------|-----------------------|------------------ - Rancher v2.5 | Benchmark v1.5 | Kubernetes 1.15 - -[Click here to download a PDF version of this document](https://releases.rancher.com/documents/security/2.5/Rancher_Hardening_Guide_CIS_1.5.pdf) - -### Overview - -This document provides prescriptive guidance for hardening a RKE cluster to be used for installing Rancher v2.5 with Kubernetes v1.15 or provisioning a RKE cluster with Kubernetes 1.15 to be used within Rancher v2.5. It outlines the configurations required to address Kubernetes benchmark controls from the Center for Information Security (CIS). - -For more detail about evaluating a hardened cluster against the official CIS benchmark, refer to the [CIS 1.5 Benchmark - Self-Assessment Guide - Rancher v2.5]({{< baseurl >}}/rancher/v2.6/en/security/rancher-2.5/1.5-benchmark-2.5/). - -#### Known Issues - -- Rancher **exec shell** and **view logs** for pods are **not** functional in a CIS 1.5 hardened setup when only public IP is provided when registering custom nodes. This functionality requires a private IP to be provided when registering the custom nodes. -- When setting the `default_pod_security_policy_template_id:` to `restricted` Rancher creates **RoleBindings** and **ClusterRoleBindings** on the default service accounts. The CIS 1.5 5.1.5 check requires the default service accounts have no roles or cluster roles bound to it apart from the defaults. In addition the default service accounts should be configured such that it does not provide a service account token and does not have any explicit rights assignments. - -### Configure Kernel Runtime Parameters - -The following `sysctl` configuration is recommended for all nodes type in the cluster. Set the following parameters in `/etc/sysctl.d/90-kubelet.conf`: - -``` -vm.overcommit_memory=1 -vm.panic_on_oom=0 -kernel.panic=10 -kernel.panic_on_oops=1 -kernel.keys.root_maxbytes=25000000 -``` - -Run `sysctl -p /etc/sysctl.d/90-kubelet.conf` to enable the settings. - -### Configure `etcd` user and group -A user account and group for the **etcd** service is required to be setup before installing RKE. The **uid** and **gid** for the **etcd** user will be used in the RKE **config.yml** to set the proper permissions for files and directories during installation time. - -#### create `etcd` user and group -To create the **etcd** group run the following console commands. - -The commands below use `52034` for **uid** and **gid** are for example purposes. Any valid unused **uid** or **gid** could also be used in lieu of `52034`. - -``` -groupadd --gid 52034 etcd -useradd --comment "etcd service account" --uid 52034 --gid 52034 etcd -``` - -Update the RKE **config.yml** with the **uid** and **gid** of the **etcd** user: - -``` yaml -services: - etcd: - gid: 52034 - uid: 52034 -``` - -#### Set `automountServiceAccountToken` to `false` for `default` service accounts -Kubernetes provides a default service account which is used by cluster workloads where no specific service account is assigned to the pod. Where access to the Kubernetes API from a pod is required, a specific service account should be created for that pod, and rights granted to that service account. The default service account should be configured such that it does not provide a service account token and does not have any explicit rights assignments. - -For each namespace including **default** and **kube-system** on a standard RKE install the **default** service account must include this value: - -``` -automountServiceAccountToken: false -``` - -Save the following yaml to a file called `account_update.yaml` - -``` yaml -apiVersion: v1 -kind: ServiceAccount -metadata: - name: default -automountServiceAccountToken: false -``` - -Create a bash script file called `account_update.sh`. Be sure to `chmod +x account_update.sh` so the script has execute permissions. - -``` -#!/bin/bash -e - -for namespace in $(kubectl get namespaces -A -o json | jq -r '.items[].metadata.name'); do - kubectl patch serviceaccount default -n ${namespace} -p "$(cat account_update.yaml)" -done -``` - -### Ensure that all Namespaces have Network Policies defined - -Running different applications on the same Kubernetes cluster creates a risk of one -compromised application attacking a neighboring application. Network segmentation is -important to ensure that containers can communicate only with those they are supposed -to. A network policy is a specification of how selections of pods are allowed to -communicate with each other and other network endpoints. - -Network Policies are namespace scoped. When a network policy is introduced to a given -namespace, all traffic not allowed by the policy is denied. However, if there are no network -policies in a namespace all traffic will be allowed into and out of the pods in that -namespace. To enforce network policies, a CNI (container network interface) plugin must be enabled. -This guide uses [canal](https://github.com/projectcalico/canal) to provide the policy enforcement. -Additional information about CNI providers can be found -[here](https://rancher.com/blog/2019/2019-03-21-comparing-kubernetes-cni-providers-flannel-calico-canal-and-weave/) - -Once a CNI provider is enabled on a cluster a default network policy can be applied. For reference purposes a -**permissive** example is provide below. If you want to allow all traffic to all pods in a namespace -(even if policies are added that cause some pods to be treated as “isolated”), -you can create a policy that explicitly allows all traffic in that namespace. Save the following `yaml` as -`default-allow-all.yaml`. Additional [documentation](https://kubernetes.io/docs/concepts/services-networking/network-policies/) -about network policies can be found on the Kubernetes site. - -> This `NetworkPolicy` is not recommended for production use - -``` yaml ---- -apiVersion: networking.k8s.io/v1 -kind: NetworkPolicy -metadata: - name: default-allow-all -spec: - podSelector: {} - ingress: - - {} - egress: - - {} - policyTypes: - - Ingress - - Egress -``` - -Create a bash script file called `apply_networkPolicy_to_all_ns.sh`. Be sure to -`chmod +x apply_networkPolicy_to_all_ns.sh` so the script has execute permissions. - -``` -#!/bin/bash -e - -for namespace in $(kubectl get namespaces -A -o json | jq -r '.items[].metadata.name'); do - kubectl apply -f default-allow-all.yaml -n ${namespace} -done -``` -Execute this script to apply the `default-allow-all.yaml` the **permissive** `NetworkPolicy` to all namespaces. - -### Reference Hardened RKE `cluster.yml` configuration - -The reference `cluster.yml` is used by the RKE CLI that provides the configuration needed to achieve a hardened install -of Rancher Kubernetes Engine (RKE). Install [documentation](https://rancher.com/docs/rke/latest/en/installation/) is -provided with additional details about the configuration items. This reference `cluster.yml` does not include the required **nodes** directive which will vary depending on your environment. Documentation for node configuration can be found here: https://rancher.com/docs/rke/latest/en/config-options/nodes - - -``` yaml -# If you intend to deploy Kubernetes in an air-gapped environment, -# please consult the documentation on how to configure custom RKE images. -kubernetes_version: "v1.15.9-rancher1-1" -enable_network_policy: true -default_pod_security_policy_template_id: "restricted" -# the nodes directive is required and will vary depending on your environment -# documentation for node configuration can be found here: -# https://rancher.com/docs/rke/latest/en/config-options/nodes -nodes: -services: - etcd: - uid: 52034 - gid: 52034 - kube-api: - pod_security_policy: true - secrets_encryption_config: - enabled: true - audit_log: - enabled: true - admission_configuration: - event_rate_limit: - enabled: true - kube-controller: - extra_args: - feature-gates: "RotateKubeletServerCertificate=true" - scheduler: - image: "" - extra_args: {} - extra_binds: [] - extra_env: [] - kubelet: - generate_serving_certificate: true - extra_args: - feature-gates: "RotateKubeletServerCertificate=true" - protect-kernel-defaults: "true" - tls-cipher-suites: "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256" - extra_binds: [] - extra_env: [] - cluster_domain: "" - infra_container_image: "" - cluster_dns_server: "" - fail_swap_on: false - kubeproxy: - image: "" - extra_args: {} - extra_binds: [] - extra_env: [] -network: - plugin: "" - options: {} - mtu: 0 - node_selector: {} -authentication: - strategy: "" - sans: [] - webhook: null -addons: | - --- - apiVersion: v1 - kind: Namespace - metadata: - name: ingress-nginx - --- - apiVersion: rbac.authorization.k8s.io/v1 - kind: Role - metadata: - name: default-psp-role - namespace: ingress-nginx - rules: - - apiGroups: - - extensions - resourceNames: - - default-psp - resources: - - podsecuritypolicies - verbs: - - use - --- - apiVersion: rbac.authorization.k8s.io/v1 - kind: RoleBinding - metadata: - name: default-psp-rolebinding - namespace: ingress-nginx - roleRef: - apiGroup: rbac.authorization.k8s.io - kind: Role - name: default-psp-role - subjects: - - apiGroup: rbac.authorization.k8s.io - kind: Group - name: system:serviceaccounts - - apiGroup: rbac.authorization.k8s.io - kind: Group - name: system:authenticated - --- - apiVersion: v1 - kind: Namespace - metadata: - name: cattle-system - --- - apiVersion: rbac.authorization.k8s.io/v1 - kind: Role - metadata: - name: default-psp-role - namespace: cattle-system - rules: - - apiGroups: - - extensions - resourceNames: - - default-psp - resources: - - podsecuritypolicies - verbs: - - use - --- - apiVersion: rbac.authorization.k8s.io/v1 - kind: RoleBinding - metadata: - name: default-psp-rolebinding - namespace: cattle-system - roleRef: - apiGroup: rbac.authorization.k8s.io - kind: Role - name: default-psp-role - subjects: - - apiGroup: rbac.authorization.k8s.io - kind: Group - name: system:serviceaccounts - - apiGroup: rbac.authorization.k8s.io - kind: Group - name: system:authenticated - --- - apiVersion: policy/v1beta1 - kind: PodSecurityPolicy - metadata: - name: restricted - spec: - requiredDropCapabilities: - - NET_RAW - privileged: false - allowPrivilegeEscalation: false - defaultAllowPrivilegeEscalation: false - fsGroup: - rule: RunAsAny - runAsUser: - rule: MustRunAsNonRoot - seLinux: - rule: RunAsAny - supplementalGroups: - rule: RunAsAny - volumes: - - emptyDir - - secret - - persistentVolumeClaim - - downwardAPI - - configMap - - projected - --- - apiVersion: rbac.authorization.k8s.io/v1 - kind: ClusterRole - metadata: - name: psp:restricted - rules: - - apiGroups: - - extensions - resourceNames: - - restricted - resources: - - podsecuritypolicies - verbs: - - use - --- - apiVersion: rbac.authorization.k8s.io/v1 - kind: ClusterRoleBinding - metadata: - name: psp:restricted - roleRef: - apiGroup: rbac.authorization.k8s.io - kind: ClusterRole - name: psp:restricted - subjects: - - apiGroup: rbac.authorization.k8s.io - kind: Group - name: system:serviceaccounts - - apiGroup: rbac.authorization.k8s.io - kind: Group - name: system:authenticated - --- - apiVersion: v1 - kind: ServiceAccount - metadata: - name: tiller - namespace: kube-system - --- - apiVersion: rbac.authorization.k8s.io/v1 - kind: ClusterRoleBinding - metadata: - name: tiller - roleRef: - apiGroup: rbac.authorization.k8s.io - kind: ClusterRole - name: cluster-admin - subjects: - - kind: ServiceAccount - name: tiller - namespace: kube-system - -addons_include: [] -system_images: - etcd: "" - alpine: "" - nginx_proxy: "" - cert_downloader: "" - kubernetes_services_sidecar: "" - kubedns: "" - dnsmasq: "" - kubedns_sidecar: "" - kubedns_autoscaler: "" - coredns: "" - coredns_autoscaler: "" - kubernetes: "" - flannel: "" - flannel_cni: "" - calico_node: "" - calico_cni: "" - calico_controllers: "" - calico_ctl: "" - calico_flexvol: "" - canal_node: "" - canal_cni: "" - canal_flannel: "" - canal_flexvol: "" - weave_node: "" - weave_cni: "" - pod_infra_container: "" - ingress: "" - ingress_backend: "" - metrics_server: "" - windows_pod_infra_container: "" -ssh_key_path: "" -ssh_cert_path: "" -ssh_agent_auth: false -authorization: - mode: "" - options: {} -ignore_docker_version: false -private_registries: [] -ingress: - provider: "" - options: {} - node_selector: {} - extra_args: {} - dns_policy: "" - extra_envs: [] - extra_volumes: [] - extra_volume_mounts: [] -cluster_name: "" -prefix_path: "" -addon_job_timeout: 0 -bastion_host: - address: "" - port: "" - user: "" - ssh_key: "" - ssh_key_path: "" - ssh_cert: "" - ssh_cert_path: "" -monitoring: - provider: "" - options: {} - node_selector: {} -restore: - restore: false - snapshot_name: "" -dns: null -``` - -### Reference Hardened RKE Template configuration - -The reference RKE Template provides the configuration needed to achieve a hardened install of Kubenetes. -RKE Templates are used to provision Kubernetes and define Rancher settings. Follow the Rancher -[documentaion](https://rancher.com/docs/rancher/v2.6/en/installation) for additional installation and RKE Template details. - -``` yaml -# -# Cluster Config -# -default_pod_security_policy_template_id: restricted -docker_root_dir: /var/lib/docker -enable_cluster_alerting: false -enable_cluster_monitoring: false -enable_network_policy: true -# -# Rancher Config -# -rancher_kubernetes_engine_config: - addon_job_timeout: 30 - addons: |- - --- - apiVersion: v1 - kind: Namespace - metadata: - name: ingress-nginx - --- - apiVersion: rbac.authorization.k8s.io/v1 - kind: Role - metadata: - name: default-psp-role - namespace: ingress-nginx - rules: - - apiGroups: - - extensions - resourceNames: - - default-psp - resources: - - podsecuritypolicies - verbs: - - use - --- - apiVersion: rbac.authorization.k8s.io/v1 - kind: RoleBinding - metadata: - name: default-psp-rolebinding - namespace: ingress-nginx - roleRef: - apiGroup: rbac.authorization.k8s.io - kind: Role - name: default-psp-role - subjects: - - apiGroup: rbac.authorization.k8s.io - kind: Group - name: system:serviceaccounts - - apiGroup: rbac.authorization.k8s.io - kind: Group - name: system:authenticated - --- - apiVersion: v1 - kind: Namespace - metadata: - name: cattle-system - --- - apiVersion: rbac.authorization.k8s.io/v1 - kind: Role - metadata: - name: default-psp-role - namespace: cattle-system - rules: - - apiGroups: - - extensions - resourceNames: - - default-psp - resources: - - podsecuritypolicies - verbs: - - use - --- - apiVersion: rbac.authorization.k8s.io/v1 - kind: RoleBinding - metadata: - name: default-psp-rolebinding - namespace: cattle-system - roleRef: - apiGroup: rbac.authorization.k8s.io - kind: Role - name: default-psp-role - subjects: - - apiGroup: rbac.authorization.k8s.io - kind: Group - name: system:serviceaccounts - - apiGroup: rbac.authorization.k8s.io - kind: Group - name: system:authenticated - --- - apiVersion: policy/v1beta1 - kind: PodSecurityPolicy - metadata: - name: restricted - spec: - requiredDropCapabilities: - - NET_RAW - privileged: false - allowPrivilegeEscalation: false - defaultAllowPrivilegeEscalation: false - fsGroup: - rule: RunAsAny - runAsUser: - rule: MustRunAsNonRoot - seLinux: - rule: RunAsAny - supplementalGroups: - rule: RunAsAny - volumes: - - emptyDir - - secret - - persistentVolumeClaim - - downwardAPI - - configMap - - projected - --- - apiVersion: rbac.authorization.k8s.io/v1 - kind: ClusterRole - metadata: - name: psp:restricted - rules: - - apiGroups: - - extensions - resourceNames: - - restricted - resources: - - podsecuritypolicies - verbs: - - use - --- - apiVersion: rbac.authorization.k8s.io/v1 - kind: ClusterRoleBinding - metadata: - name: psp:restricted - roleRef: - apiGroup: rbac.authorization.k8s.io - kind: ClusterRole - name: psp:restricted - subjects: - - apiGroup: rbac.authorization.k8s.io - kind: Group - name: system:serviceaccounts - - apiGroup: rbac.authorization.k8s.io - kind: Group - name: system:authenticated - --- - apiVersion: v1 - kind: ServiceAccount - metadata: - name: tiller - namespace: kube-system - --- - apiVersion: rbac.authorization.k8s.io/v1 - kind: ClusterRoleBinding - metadata: - name: tiller - roleRef: - apiGroup: rbac.authorization.k8s.io - kind: ClusterRole - name: cluster-admin - subjects: - - kind: ServiceAccount - name: tiller - namespace: kube-system - ignore_docker_version: true - kubernetes_version: v1.15.9-rancher1-1 -# -# If you are using calico on AWS -# -# network: -# plugin: calico -# calico_network_provider: -# cloud_provider: aws -# -# # To specify flannel interface -# -# network: -# plugin: flannel -# flannel_network_provider: -# iface: eth1 -# -# # To specify flannel interface for canal plugin -# -# network: -# plugin: canal -# canal_network_provider: -# iface: eth1 -# - network: - mtu: 0 - plugin: canal -# -# services: -# kube-api: -# service_cluster_ip_range: 10.43.0.0/16 -# kube-controller: -# cluster_cidr: 10.42.0.0/16 -# service_cluster_ip_range: 10.43.0.0/16 -# kubelet: -# cluster_domain: cluster.local -# cluster_dns_server: 10.43.0.10 -# - services: - etcd: - backup_config: - enabled: false - interval_hours: 12 - retention: 6 - safe_timestamp: false - creation: 12h - extra_args: - election-timeout: '5000' - heartbeat-interval: '500' - gid: 52034 - retention: 72h - snapshot: false - uid: 52034 - kube_api: - always_pull_images: false - audit_log: - enabled: true - event_rate_limit: - enabled: true - pod_security_policy: true - secrets_encryption_config: - enabled: true - service_node_port_range: 30000-32767 - kube_controller: - extra_args: - address: 127.0.0.1 - feature-gates: RotateKubeletServerCertificate=true - profiling: 'false' - terminated-pod-gc-threshold: '1000' - kubelet: - extra_args: - anonymous-auth: 'false' - event-qps: '0' - feature-gates: RotateKubeletServerCertificate=true - make-iptables-util-chains: 'true' - protect-kernel-defaults: 'true' - streaming-connection-idle-timeout: 1800s - tls-cipher-suites: >- - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 - fail_swap_on: false - generate_serving_certificate: true - scheduler: - extra_args: - address: 127.0.0.1 - profiling: 'false' - ssh_agent_auth: false -windows_prefered_cluster: false -``` - -### Hardened Reference Ubuntu 18.04 LTS **cloud-config**: - -The reference **cloud-config** is generally used in cloud infrastructure environments to allow for -configuration management of compute instances. The reference config configures Ubuntu operating system level settings -needed before installing kubernetes. - -``` yaml -#cloud-config -packages: - - curl - - jq -runcmd: - - sysctl -w vm.overcommit_memory=1 - - sysctl -w kernel.panic=10 - - sysctl -w kernel.panic_on_oops=1 - - curl https://releases.rancher.com/install-docker/18.09.sh | sh - - usermod -aG docker ubuntu - - return=1; while [ $return != 0 ]; do sleep 2; docker ps; return=$?; done - - addgroup --gid 52034 etcd - - useradd --comment "etcd service account" --uid 52034 --gid 52034 etcd -write_files: - - path: /etc/sysctl.d/kubelet.conf - owner: root:root - permissions: "0644" - content: | - vm.overcommit_memory=1 - kernel.panic=10 - kernel.panic_on_oops=1 -``` diff --git a/content/rancher/v2.6/en/security/rancher-2.5/1.6-benchmark-2.5/_index.md b/content/rancher/v2.6/en/security/rancher-2.5/1.6-benchmark-2.5/_index.md deleted file mode 100644 index d7803779eb6..00000000000 --- a/content/rancher/v2.6/en/security/rancher-2.5/1.6-benchmark-2.5/_index.md +++ /dev/null @@ -1,3317 +0,0 @@ ---- -title: CIS 1.6 Benchmark - Self-Assessment Guide - Rancher v2.5.4 -weight: 101 ---- - -### CIS 1.6 Kubernetes Benchmark - Rancher v2.5.4 with Kubernetes v1.18 - -[Click here to download a PDF version of this document](https://releases.rancher.com/documents/security/2.5/Rancher_1.6_Benchmark_Assessment.pdf) - -#### Overview - -This document is a companion to the Rancher v2.5.4 security hardening guide. The hardening guide provides prescriptive guidance for hardening a production installation of Rancher, and this benchmark guide is meant to help you evaluate the level of security of the hardened cluster against each control in the benchmark. - -This guide corresponds to specific versions of the hardening guide, Rancher, CIS Benchmark, and Kubernetes: - -Hardening Guide Version | Rancher Version | CIS Benchmark Version | Kubernetes Version ----------------------------|----------|---------|------- -Hardening Guide with CIS 1.6 Benchmark | Rancher v2.5.4 | CIS 1.6| Kubernetes v1.18 - -Because Rancher and RKE install Kubernetes services as Docker containers, many of the control verification checks in the CIS Kubernetes Benchmark don't apply and will have a result of `Not Applicable`. This guide will walk through the various controls and provide updated example commands to audit compliance in Rancher-created clusters. - -This document is to be used by Rancher operators, security teams, auditors and decision makers. - -For more detail about each audit, including rationales and remediations for failing tests, you can refer to the corresponding section of the CIS Kubernetes Benchmark 1.6. You can download the benchmark after logging in to [CISecurity.org]( https://www.cisecurity.org/benchmark/kubernetes/). - -#### Testing controls methodology - -Rancher and RKE install Kubernetes services via Docker containers. Configuration is defined by arguments passed to the container at the time of initialization, not via configuration files. - -Where control audits differ from the original CIS benchmark, the audit commands specific to Rancher Labs are provided for testing. -When performing the tests, you will need access to the Docker command line on the hosts of all three RKE roles. The commands also make use of the the [jq](https://stedolan.github.io/jq/) and [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/) (with valid config) tools to and are required in the testing and evaluation of test results. - -### Controls - -## 1.1 Etcd Node Configuration Files -### 1.1.11 Ensure that the etcd data directory permissions are set to 700 or more restrictive (Automated) - -**Result:** pass - -**Remediation:** -On the etcd server node, get the etcd data directory, passed as an argument --data-dir, -from the below command: -ps -ef | grep etcd Run the below command (based on the etcd data directory found above). For example, -chmod 700 /var/lib/etcd - - -**Audit:** - -```bash -stat -c %a /node/var/lib/etcd -``` - -**Expected Result**: - -```console -'700' is equal to '700' -``` - -**Returned Value**: - -```console -700 - -``` -### 1.1.12 Ensure that the etcd data directory ownership is set to etcd:etcd (Automated) - -**Result:** pass - -**Remediation:** -On the etcd server node, get the etcd data directory, passed as an argument --data-dir, -from the below command: -ps -ef | grep etcd -Run the below command (based on the etcd data directory found above). -For example, chown etcd:etcd /var/lib/etcd - -A system service account is required for etcd data directory ownership. -Refer to Rancher's hardening guide for more details on how to configure this ownership. - - -**Audit:** - -```bash -stat -c %U:%G /node/var/lib/etcd -``` - -**Expected Result**: - -```console -'etcd:etcd' is present -``` - -**Returned Value**: - -```console -etcd:etcd - -``` -### 1.1.19 Ensure that the Kubernetes PKI directory and file ownership is set to root:root (Automated) - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the master node. -For example, -chown -R root:root /etc/kubernetes/pki/ - - -**Audit:** - -```bash -check_files_owner_in_dir.sh /node/etc/kubernetes/ssl -``` - -**Expected Result**: - -```console -'true' is equal to 'true' -``` - -**Audit Script:** -```bash -#!/usr/bin/env bash - -# This script is used to ensure the owner is set to root:root for -# the given directory and all the files in it -# -# inputs: -# $1 = /full/path/to/directory -# -# outputs: -# true/false - -INPUT_DIR=$1 - -if [[ "${INPUT_DIR}" == "" ]]; then - echo "false" - exit -fi - -if [[ $(stat -c %U:%G ${INPUT_DIR}) != "root:root" ]]; then - echo "false" - exit -fi - -statInfoLines=$(stat -c "%n %U:%G" ${INPUT_DIR}/*) -while read -r statInfoLine; do - f=$(echo ${statInfoLine} | cut -d' ' -f1) - p=$(echo ${statInfoLine} | cut -d' ' -f2) - - if [[ $(basename "$f" .pem) == "kube-etcd-"* ]]; then - if [[ "$p" != "root:root" && "$p" != "etcd:etcd" ]]; then - echo "false" - exit - fi - else - if [[ "$p" != "root:root" ]]; then - echo "false" - exit - fi - fi -done <<< "${statInfoLines}" - - -echo "true" -exit - -``` -**Returned Value**: - -```console -true - -``` -### 1.1.20 Ensure that the Kubernetes PKI certificate file permissions are set to 644 or more restrictive (Automated) - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the master node. -For example, -chmod -R 644 /etc/kubernetes/pki/*.crt - - -**Audit:** - -```bash -check_files_permissions.sh /node/etc/kubernetes/ssl/!(*key).pem -``` - -**Expected Result**: - -```console -'true' is equal to 'true' -``` - -**Audit Script:** -```bash -#!/usr/bin/env bash - -# This script is used to ensure the file permissions are set to 644 or -# more restrictive for all files in a given directory or a wildcard -# selection of files -# -# inputs: -# $1 = /full/path/to/directory or /path/to/fileswithpattern -# ex: !(*key).pem -# -# $2 (optional) = permission (ex: 600) -# -# outputs: -# true/false - -# Turn on "extended glob" for use of '!' in wildcard -shopt -s extglob - -# Turn off history to avoid surprises when using '!' -set -H - -USER_INPUT=$1 - -if [[ "${USER_INPUT}" == "" ]]; then - echo "false" - exit -fi - - -if [[ -d ${USER_INPUT} ]]; then - PATTERN="${USER_INPUT}/*" -else - PATTERN="${USER_INPUT}" -fi - -PERMISSION="" -if [[ "$2" != "" ]]; then - PERMISSION=$2 -fi - -FILES_PERMISSIONS=$(stat -c %n\ %a ${PATTERN}) - -while read -r fileInfo; do - p=$(echo ${fileInfo} | cut -d' ' -f2) - - if [[ "${PERMISSION}" != "" ]]; then - if [[ "$p" != "${PERMISSION}" ]]; then - echo "false" - exit - fi - else - if [[ "$p" != "644" && "$p" != "640" && "$p" != "600" ]]; then - echo "false" - exit - fi - fi -done <<< "${FILES_PERMISSIONS}" - - -echo "true" -exit - -``` -**Returned Value**: - -```console -true - -``` -### 1.1.21 Ensure that the Kubernetes PKI key file permissions are set to 600 (Automated) - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the master node. -For example, -chmod -R 600 /etc/kubernetes/ssl/*key.pem - - -**Audit:** - -```bash -check_files_permissions.sh /node/etc/kubernetes/ssl/*key.pem 600 -``` - -**Expected Result**: - -```console -'true' is equal to 'true' -``` - -**Audit Script:** -```bash -#!/usr/bin/env bash - -# This script is used to ensure the file permissions are set to 644 or -# more restrictive for all files in a given directory or a wildcard -# selection of files -# -# inputs: -# $1 = /full/path/to/directory or /path/to/fileswithpattern -# ex: !(*key).pem -# -# $2 (optional) = permission (ex: 600) -# -# outputs: -# true/false - -# Turn on "extended glob" for use of '!' in wildcard -shopt -s extglob - -# Turn off history to avoid surprises when using '!' -set -H - -USER_INPUT=$1 - -if [[ "${USER_INPUT}" == "" ]]; then - echo "false" - exit -fi - - -if [[ -d ${USER_INPUT} ]]; then - PATTERN="${USER_INPUT}/*" -else - PATTERN="${USER_INPUT}" -fi - -PERMISSION="" -if [[ "$2" != "" ]]; then - PERMISSION=$2 -fi - -FILES_PERMISSIONS=$(stat -c %n\ %a ${PATTERN}) - -while read -r fileInfo; do - p=$(echo ${fileInfo} | cut -d' ' -f2) - - if [[ "${PERMISSION}" != "" ]]; then - if [[ "$p" != "${PERMISSION}" ]]; then - echo "false" - exit - fi - else - if [[ "$p" != "644" && "$p" != "640" && "$p" != "600" ]]; then - echo "false" - exit - fi - fi -done <<< "${FILES_PERMISSIONS}" - - -echo "true" -exit - -``` -**Returned Value**: - -```console -true - -``` -### 1.1.1 Ensure that the API server pod specification file permissions are set to 644 or more restrictive (Automated) - -**Result:** notApplicable - -**Remediation:** -Cluster provisioned by RKE doesn't require or maintain a configuration file for kube-apiserver. -All configuration is passed in as arguments at container run time. - - -**Audit:** - -```bash -/bin/sh -c 'if test -e /etc/kubernetes/manifests/kube-apiserver.yaml; then stat -c permissions=%a /etc/kubernetes/manifests/kube-apiserver.yaml; fi' -``` - - -### 1.1.2 Ensure that the API server pod specification file ownership is set to root:root (Automated) - -**Result:** notApplicable - -**Remediation:** -Cluster provisioned by RKE doesn't require or maintain a configuration file for kube-apiserver. -All configuration is passed in as arguments at container run time. - - -**Audit:** - -```bash -/bin/sh -c 'if test -e /etc/kubernetes/manifests/kube-apiserver.yaml; then stat -c %U:%G /etc/kubernetes/manifests/kube-apiserver.yaml; fi' -``` - - -### 1.1.3 Ensure that the controller manager pod specification file permissions are set to 644 or more restrictive (Automated) - -**Result:** notApplicable - -**Remediation:** -Cluster provisioned by RKE doesn't require or maintain a configuration file for controller-manager. -All configuration is passed in as arguments at container run time. - - -**Audit:** - -```bash -/bin/sh -c 'if test -e /etc/kubernetes/manifests/kube-controller-manager.yaml; then stat -c permissions=%a /etc/kubernetes/manifests/kube-controller-manager.yaml; fi' -``` - - -### 1.1.4 Ensure that the controller manager pod specification file ownership is set to root:root (Automated) - -**Result:** notApplicable - -**Remediation:** -Cluster provisioned by RKE doesn't require or maintain a configuration file for controller-manager. -All configuration is passed in as arguments at container run time. - - -**Audit:** - -```bash -/bin/sh -c 'if test -e /etc/kubernetes/manifests/kube-controller-manager.yaml; then stat -c %U:%G /etc/kubernetes/manifests/kube-controller-manager.yaml; fi' -``` - - -### 1.1.5 Ensure that the scheduler pod specification file permissions are set to 644 or more restrictive (Automated) - -**Result:** notApplicable - -**Remediation:** -Cluster provisioned by RKE doesn't require or maintain a configuration file for scheduler. -All configuration is passed in as arguments at container run time. - - -**Audit:** - -```bash -/bin/sh -c 'if test -e /etc/kubernetes/manifests/kube-scheduler.yaml; then stat -c permissions=%a /etc/kubernetes/manifests/kube-scheduler.yaml; fi' -``` - - -### 1.1.6 Ensure that the scheduler pod specification file ownership is set to root:root (Automated) - -**Result:** notApplicable - -**Remediation:** -Cluster provisioned by RKE doesn't require or maintain a configuration file for scheduler. -All configuration is passed in as arguments at container run time. - - -**Audit:** - -```bash -/bin/sh -c 'if test -e /etc/kubernetes/manifests/kube-scheduler.yaml; then stat -c %U:%G /etc/kubernetes/manifests/kube-scheduler.yaml; fi' -``` - - -### 1.1.7 Ensure that the etcd pod specification file permissions are set to 644 or more restrictive (Automated) - -**Result:** notApplicable - -**Remediation:** -Cluster provisioned by RKE doesn't require or maintain a configuration file for etcd. -All configuration is passed in as arguments at container run time. - - -**Audit:** - -```bash -/bin/sh -c 'if test -e /etc/kubernetes/manifests/etcd.yaml; then stat -c permissions=%a /etc/kubernetes/manifests/etcd.yaml; fi' -``` - - -### 1.1.8 Ensure that the etcd pod specification file ownership is set to root:root (Automated) - -**Result:** notApplicable - -**Remediation:** -Cluster provisioned by RKE doesn't require or maintain a configuration file for etcd. -All configuration is passed in as arguments at container run time. - - -**Audit:** - -```bash -/bin/sh -c 'if test -e /etc/kubernetes/manifests/etcd.yaml; then stat -c %U:%G /etc/kubernetes/manifests/etcd.yaml; fi' -``` - - -### 1.1.9 Ensure that the Container Network Interface file permissions are set to 644 or more restrictive (Manual) - -**Result:** warn - -**Remediation:** -Run the below command (based on the file location on your system) on the master node. -For example, -chmod 644 - - -**Audit:** - -```bash -stat -c permissions=%a -``` - - -### 1.1.10 Ensure that the Container Network Interface file ownership is set to root:root (Manual) - -**Result:** warn - -**Remediation:** -Run the below command (based on the file location on your system) on the master node. -For example, -chown root:root - - -**Audit:** - -```bash -stat -c %U:%G -``` - - -### 1.1.13 Ensure that the admin.conf file permissions are set to 644 or more restrictive (Automated) - -**Result:** notApplicable - -**Remediation:** -Cluster provisioned by RKE does not store the kubernetes default kubeconfig credentials file on the nodes. - - -**Audit:** - -```bash -/bin/sh -c 'if test -e /etc/kubernetes/admin.conf; then stat -c permissions=%a /etc/kubernetes/admin.conf; fi' -``` - - -### 1.1.14 Ensure that the admin.conf file ownership is set to root:root (Automated) - -**Result:** notApplicable - -**Remediation:** -Cluster provisioned by RKE does not store the kubernetes default kubeconfig credentials file on the nodes. - - -**Audit:** - -```bash -/bin/sh -c 'if test -e /etc/kubernetes/admin.conf; then stat -c %U:%G /etc/kubernetes/admin.conf; fi' -``` - - -### 1.1.15 Ensure that the scheduler.conf file permissions are set to 644 or more restrictive (Automated) - -**Result:** notApplicable - -**Remediation:** -Cluster provisioned by RKE doesn't require or maintain a configuration file for scheduler. -All configuration is passed in as arguments at container run time. - - -**Audit:** - -```bash -/bin/sh -c 'if test -e scheduler; then stat -c permissions=%a scheduler; fi' -``` - - -### 1.1.16 Ensure that the scheduler.conf file ownership is set to root:root (Automated) - -**Result:** notApplicable - -**Remediation:** -Cluster provisioned by RKE doesn't require or maintain a configuration file for scheduler. -All configuration is passed in as arguments at container run time. - - -**Audit:** - -```bash -/bin/sh -c 'if test -e scheduler; then stat -c %U:%G scheduler; fi' -``` - - -### 1.1.17 Ensure that the controller-manager.conf file permissions are set to 644 or more restrictive (Automated) - -**Result:** notApplicable - -**Remediation:** -Cluster provisioned by RKE doesn't require or maintain a configuration file for controller-manager. -All configuration is passed in as arguments at container run time. - - -**Audit:** - -```bash -/bin/sh -c 'if test -e controllermanager; then stat -c permissions=%a controllermanager; fi' -``` - - -### 1.1.18 Ensure that the controller-manager.conf file ownership is set to root:root (Automated) - -**Result:** notApplicable - -**Remediation:** -Cluster provisioned by RKE doesn't require or maintain a configuration file for controller-manager. -All configuration is passed in as arguments at container run time. - - -**Audit:** - -```bash -/bin/sh -c 'if test -e controllermanager; then stat -c %U:%G controllermanager; fi' -``` - - -## 1.2 API Server -### 1.2.1 Ensure that the --anonymous-auth argument is set to false (Automated) - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the master node and set the below parameter. ---anonymous-auth=false - - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'false' is equal to 'false' -``` - -**Returned Value**: - -```console -root 4643 4626 22 16:15 ? 00:00:46 kube-apiserver --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authorization-mode=Node,RBAC --audit-log-maxsize=100 --audit-log-format=json --requestheader-allowed-names=kube-apiserver-proxy-client --cloud-provider= --etcd-prefix=/registry --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --allow-privileged=true --service-account-lookup=true --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-policy-file=/etc/kubernetes/audit-policy.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --storage-backend=etcd3 --anonymous-auth=false --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --advertise-address=192.168.1.225 --audit-log-maxage=30 --etcd-servers=https://192.168.1.225:2379 --runtime-config=policy/v1beta1/podsecuritypolicy=true --bind-address=0.0.0.0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --insecure-port=0 --requestheader-group-headers=X-Remote-Group --secure-port=6443 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --requestheader-extra-headers-prefix=X-Remote-Extra- --profiling=false --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User - -``` -### 1.2.2 Ensure that the --basic-auth-file argument is not set (Automated) - -**Result:** pass - -**Remediation:** -Follow the documentation and configure alternate mechanisms for authentication. Then, -edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the master node and remove the --basic-auth-file= parameter. - - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--basic-auth-file' is not present -``` - -**Returned Value**: - -```console -root 4643 4626 22 16:15 ? 00:00:46 kube-apiserver --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authorization-mode=Node,RBAC --audit-log-maxsize=100 --audit-log-format=json --requestheader-allowed-names=kube-apiserver-proxy-client --cloud-provider= --etcd-prefix=/registry --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --allow-privileged=true --service-account-lookup=true --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-policy-file=/etc/kubernetes/audit-policy.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --storage-backend=etcd3 --anonymous-auth=false --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --advertise-address=192.168.1.225 --audit-log-maxage=30 --etcd-servers=https://192.168.1.225:2379 --runtime-config=policy/v1beta1/podsecuritypolicy=true --bind-address=0.0.0.0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --insecure-port=0 --requestheader-group-headers=X-Remote-Group --secure-port=6443 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --requestheader-extra-headers-prefix=X-Remote-Extra- --profiling=false --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User - -``` -### 1.2.3 Ensure that the --token-auth-file parameter is not set (Automated) - -**Result:** pass - -**Remediation:** -Follow the documentation and configure alternate mechanisms for authentication. Then, -edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the master node and remove the --token-auth-file= parameter. - - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--token-auth-file' is not present -``` - -**Returned Value**: - -```console -root 4643 4626 22 16:15 ? 00:00:46 kube-apiserver --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authorization-mode=Node,RBAC --audit-log-maxsize=100 --audit-log-format=json --requestheader-allowed-names=kube-apiserver-proxy-client --cloud-provider= --etcd-prefix=/registry --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --allow-privileged=true --service-account-lookup=true --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-policy-file=/etc/kubernetes/audit-policy.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --storage-backend=etcd3 --anonymous-auth=false --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --advertise-address=192.168.1.225 --audit-log-maxage=30 --etcd-servers=https://192.168.1.225:2379 --runtime-config=policy/v1beta1/podsecuritypolicy=true --bind-address=0.0.0.0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --insecure-port=0 --requestheader-group-headers=X-Remote-Group --secure-port=6443 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --requestheader-extra-headers-prefix=X-Remote-Extra- --profiling=false --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User - -``` -### 1.2.4 Ensure that the --kubelet-https argument is set to true (Automated) - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the master node and remove the --kubelet-https parameter. - - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--kubelet-https' is not present OR '--kubelet-https' is not present -``` - -**Returned Value**: - -```console -root 4643 4626 22 16:15 ? 00:00:46 kube-apiserver --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authorization-mode=Node,RBAC --audit-log-maxsize=100 --audit-log-format=json --requestheader-allowed-names=kube-apiserver-proxy-client --cloud-provider= --etcd-prefix=/registry --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --allow-privileged=true --service-account-lookup=true --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-policy-file=/etc/kubernetes/audit-policy.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --storage-backend=etcd3 --anonymous-auth=false --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --advertise-address=192.168.1.225 --audit-log-maxage=30 --etcd-servers=https://192.168.1.225:2379 --runtime-config=policy/v1beta1/podsecuritypolicy=true --bind-address=0.0.0.0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --insecure-port=0 --requestheader-group-headers=X-Remote-Group --secure-port=6443 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --requestheader-extra-headers-prefix=X-Remote-Extra- --profiling=false --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User - -``` -### 1.2.5 Ensure that the --kubelet-client-certificate and --kubelet-client-key arguments are set as appropriate (Automated) - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection between the -apiserver and kubelets. Then, edit API server pod specification file -/etc/kubernetes/manifests/kube-apiserver.yaml on the master node and set the -kubelet client certificate and key parameters as below. ---kubelet-client-certificate= ---kubelet-client-key= - - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--kubelet-client-certificate' is present AND '--kubelet-client-key' is present -``` - -**Returned Value**: - -```console -root 4643 4626 22 16:15 ? 00:00:46 kube-apiserver --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authorization-mode=Node,RBAC --audit-log-maxsize=100 --audit-log-format=json --requestheader-allowed-names=kube-apiserver-proxy-client --cloud-provider= --etcd-prefix=/registry --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --allow-privileged=true --service-account-lookup=true --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-policy-file=/etc/kubernetes/audit-policy.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --storage-backend=etcd3 --anonymous-auth=false --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --advertise-address=192.168.1.225 --audit-log-maxage=30 --etcd-servers=https://192.168.1.225:2379 --runtime-config=policy/v1beta1/podsecuritypolicy=true --bind-address=0.0.0.0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --insecure-port=0 --requestheader-group-headers=X-Remote-Group --secure-port=6443 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --requestheader-extra-headers-prefix=X-Remote-Extra- --profiling=false --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User - -``` -### 1.2.6 Ensure that the --kubelet-certificate-authority argument is set as appropriate (Automated) - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and setup the TLS connection between -the apiserver and kubelets. Then, edit the API server pod specification file -/etc/kubernetes/manifests/kube-apiserver.yaml on the master node and set the ---kubelet-certificate-authority parameter to the path to the cert file for the certificate authority. ---kubelet-certificate-authority= - - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--kubelet-certificate-authority' is present -``` - -**Returned Value**: - -```console -root 4643 4626 22 16:15 ? 00:00:46 kube-apiserver --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authorization-mode=Node,RBAC --audit-log-maxsize=100 --audit-log-format=json --requestheader-allowed-names=kube-apiserver-proxy-client --cloud-provider= --etcd-prefix=/registry --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --allow-privileged=true --service-account-lookup=true --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-policy-file=/etc/kubernetes/audit-policy.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --storage-backend=etcd3 --anonymous-auth=false --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --advertise-address=192.168.1.225 --audit-log-maxage=30 --etcd-servers=https://192.168.1.225:2379 --runtime-config=policy/v1beta1/podsecuritypolicy=true --bind-address=0.0.0.0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --insecure-port=0 --requestheader-group-headers=X-Remote-Group --secure-port=6443 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --requestheader-extra-headers-prefix=X-Remote-Extra- --profiling=false --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User - -``` -### 1.2.7 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated) - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the master node and set the --authorization-mode parameter to values other than AlwaysAllow. -One such example could be as below. ---authorization-mode=RBAC - - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console - 'Node,RBAC' not have 'AlwaysAllow' -``` - -**Returned Value**: - -```console -root 4643 4626 22 16:15 ? 00:00:46 kube-apiserver --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authorization-mode=Node,RBAC --audit-log-maxsize=100 --audit-log-format=json --requestheader-allowed-names=kube-apiserver-proxy-client --cloud-provider= --etcd-prefix=/registry --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --allow-privileged=true --service-account-lookup=true --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-policy-file=/etc/kubernetes/audit-policy.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --storage-backend=etcd3 --anonymous-auth=false --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --advertise-address=192.168.1.225 --audit-log-maxage=30 --etcd-servers=https://192.168.1.225:2379 --runtime-config=policy/v1beta1/podsecuritypolicy=true --bind-address=0.0.0.0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --insecure-port=0 --requestheader-group-headers=X-Remote-Group --secure-port=6443 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --requestheader-extra-headers-prefix=X-Remote-Extra- --profiling=false --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User - -``` -### 1.2.8 Ensure that the --authorization-mode argument includes Node (Automated) - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the master node and set the --authorization-mode parameter to a value that includes Node. ---authorization-mode=Node,RBAC - - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'Node,RBAC' has 'Node' -``` - -**Returned Value**: - -```console -root 4643 4626 22 16:15 ? 00:00:46 kube-apiserver --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authorization-mode=Node,RBAC --audit-log-maxsize=100 --audit-log-format=json --requestheader-allowed-names=kube-apiserver-proxy-client --cloud-provider= --etcd-prefix=/registry --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --allow-privileged=true --service-account-lookup=true --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-policy-file=/etc/kubernetes/audit-policy.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --storage-backend=etcd3 --anonymous-auth=false --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --advertise-address=192.168.1.225 --audit-log-maxage=30 --etcd-servers=https://192.168.1.225:2379 --runtime-config=policy/v1beta1/podsecuritypolicy=true --bind-address=0.0.0.0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --insecure-port=0 --requestheader-group-headers=X-Remote-Group --secure-port=6443 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --requestheader-extra-headers-prefix=X-Remote-Extra- --profiling=false --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User - -``` -### 1.2.9 Ensure that the --authorization-mode argument includes RBAC (Automated) - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the master node and set the --authorization-mode parameter to a value that includes RBAC, -for example: ---authorization-mode=Node,RBAC - - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'Node,RBAC' has 'RBAC' -``` - -**Returned Value**: - -```console -root 4643 4626 22 16:15 ? 00:00:46 kube-apiserver --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authorization-mode=Node,RBAC --audit-log-maxsize=100 --audit-log-format=json --requestheader-allowed-names=kube-apiserver-proxy-client --cloud-provider= --etcd-prefix=/registry --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --allow-privileged=true --service-account-lookup=true --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-policy-file=/etc/kubernetes/audit-policy.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --storage-backend=etcd3 --anonymous-auth=false --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --advertise-address=192.168.1.225 --audit-log-maxage=30 --etcd-servers=https://192.168.1.225:2379 --runtime-config=policy/v1beta1/podsecuritypolicy=true --bind-address=0.0.0.0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --insecure-port=0 --requestheader-group-headers=X-Remote-Group --secure-port=6443 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --requestheader-extra-headers-prefix=X-Remote-Extra- --profiling=false --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User - -``` -### 1.2.10 Ensure that the admission control plugin EventRateLimit is set (Automated) - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set the desired limits in a configuration file. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -and set the below parameters. ---enable-admission-plugins=...,EventRateLimit,... ---admission-control-config-file= - - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit' has 'EventRateLimit' -``` - -**Returned Value**: - -```console -root 4643 4626 22 16:15 ? 00:00:46 kube-apiserver --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authorization-mode=Node,RBAC --audit-log-maxsize=100 --audit-log-format=json --requestheader-allowed-names=kube-apiserver-proxy-client --cloud-provider= --etcd-prefix=/registry --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --allow-privileged=true --service-account-lookup=true --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-policy-file=/etc/kubernetes/audit-policy.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --storage-backend=etcd3 --anonymous-auth=false --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --advertise-address=192.168.1.225 --audit-log-maxage=30 --etcd-servers=https://192.168.1.225:2379 --runtime-config=policy/v1beta1/podsecuritypolicy=true --bind-address=0.0.0.0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --insecure-port=0 --requestheader-group-headers=X-Remote-Group --secure-port=6443 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --requestheader-extra-headers-prefix=X-Remote-Extra- --profiling=false --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User - -``` -### 1.2.11 Ensure that the admission control plugin AlwaysAdmit is not set (Automated) - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the master node and either remove the --enable-admission-plugins parameter, or set it to a -value that does not include AlwaysAdmit. - - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console - 'NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit' not have 'AlwaysAdmit' OR '--enable-admission-plugins' is not present -``` - -**Returned Value**: - -```console -root 4643 4626 22 16:15 ? 00:00:46 kube-apiserver --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authorization-mode=Node,RBAC --audit-log-maxsize=100 --audit-log-format=json --requestheader-allowed-names=kube-apiserver-proxy-client --cloud-provider= --etcd-prefix=/registry --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --allow-privileged=true --service-account-lookup=true --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-policy-file=/etc/kubernetes/audit-policy.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --storage-backend=etcd3 --anonymous-auth=false --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --advertise-address=192.168.1.225 --audit-log-maxage=30 --etcd-servers=https://192.168.1.225:2379 --runtime-config=policy/v1beta1/podsecuritypolicy=true --bind-address=0.0.0.0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --insecure-port=0 --requestheader-group-headers=X-Remote-Group --secure-port=6443 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --requestheader-extra-headers-prefix=X-Remote-Extra- --profiling=false --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User - -``` -### 1.2.12 Ensure that the admission control plugin AlwaysPullImages is set (Manual) - -**Result:** warn - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the master node and set the --enable-admission-plugins parameter to include -AlwaysPullImages. ---enable-admission-plugins=...,AlwaysPullImages,... - - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - - -### 1.2.13 Ensure that the admission control plugin SecurityContextDeny is set if PodSecurityPolicy is not used (Manual) - -**Result:** warn - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the master node and set the --enable-admission-plugins parameter to include -SecurityContextDeny, unless PodSecurityPolicy is already in place. ---enable-admission-plugins=...,SecurityContextDeny,... - - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - - -### 1.2.14 Ensure that the admission control plugin ServiceAccount is set (Automated) - -**Result:** pass - -**Remediation:** -Follow the documentation and create ServiceAccount objects as per your environment. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the master node and ensure that the --disable-admission-plugins parameter is set to a -value that does not include ServiceAccount. - - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--disable-admission-plugins' is not present OR '--disable-admission-plugins' is not present -``` - -**Returned Value**: - -```console -root 4643 4626 22 16:15 ? 00:00:46 kube-apiserver --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authorization-mode=Node,RBAC --audit-log-maxsize=100 --audit-log-format=json --requestheader-allowed-names=kube-apiserver-proxy-client --cloud-provider= --etcd-prefix=/registry --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --allow-privileged=true --service-account-lookup=true --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-policy-file=/etc/kubernetes/audit-policy.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --storage-backend=etcd3 --anonymous-auth=false --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --advertise-address=192.168.1.225 --audit-log-maxage=30 --etcd-servers=https://192.168.1.225:2379 --runtime-config=policy/v1beta1/podsecuritypolicy=true --bind-address=0.0.0.0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --insecure-port=0 --requestheader-group-headers=X-Remote-Group --secure-port=6443 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --requestheader-extra-headers-prefix=X-Remote-Extra- --profiling=false --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User - -``` -### 1.2.15 Ensure that the admission control plugin NamespaceLifecycle is set (Automated) - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the master node and set the --disable-admission-plugins parameter to -ensure it does not include NamespaceLifecycle. - - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--disable-admission-plugins' is not present OR '--disable-admission-plugins' is not present -``` - -**Returned Value**: - -```console -root 4643 4626 22 16:15 ? 00:00:46 kube-apiserver --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authorization-mode=Node,RBAC --audit-log-maxsize=100 --audit-log-format=json --requestheader-allowed-names=kube-apiserver-proxy-client --cloud-provider= --etcd-prefix=/registry --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --allow-privileged=true --service-account-lookup=true --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-policy-file=/etc/kubernetes/audit-policy.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --storage-backend=etcd3 --anonymous-auth=false --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --advertise-address=192.168.1.225 --audit-log-maxage=30 --etcd-servers=https://192.168.1.225:2379 --runtime-config=policy/v1beta1/podsecuritypolicy=true --bind-address=0.0.0.0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --insecure-port=0 --requestheader-group-headers=X-Remote-Group --secure-port=6443 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --requestheader-extra-headers-prefix=X-Remote-Extra- --profiling=false --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User - -``` -### 1.2.16 Ensure that the admission control plugin PodSecurityPolicy is set (Automated) - -**Result:** pass - -**Remediation:** -Follow the documentation and create Pod Security Policy objects as per your environment. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the master node and set the --enable-admission-plugins parameter to a -value that includes PodSecurityPolicy: ---enable-admission-plugins=...,PodSecurityPolicy,... -Then restart the API Server. - - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit' has 'PodSecurityPolicy' -``` - -**Returned Value**: - -```console -root 4643 4626 22 16:15 ? 00:00:46 kube-apiserver --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authorization-mode=Node,RBAC --audit-log-maxsize=100 --audit-log-format=json --requestheader-allowed-names=kube-apiserver-proxy-client --cloud-provider= --etcd-prefix=/registry --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --allow-privileged=true --service-account-lookup=true --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-policy-file=/etc/kubernetes/audit-policy.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --storage-backend=etcd3 --anonymous-auth=false --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --advertise-address=192.168.1.225 --audit-log-maxage=30 --etcd-servers=https://192.168.1.225:2379 --runtime-config=policy/v1beta1/podsecuritypolicy=true --bind-address=0.0.0.0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --insecure-port=0 --requestheader-group-headers=X-Remote-Group --secure-port=6443 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --requestheader-extra-headers-prefix=X-Remote-Extra- --profiling=false --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User - -``` -### 1.2.17 Ensure that the admission control plugin NodeRestriction is set (Automated) - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and configure NodeRestriction plug-in on kubelets. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the master node and set the --enable-admission-plugins parameter to a -value that includes NodeRestriction. ---enable-admission-plugins=...,NodeRestriction,... - - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit' has 'NodeRestriction' -``` - -**Returned Value**: - -```console -root 4643 4626 22 16:15 ? 00:00:46 kube-apiserver --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authorization-mode=Node,RBAC --audit-log-maxsize=100 --audit-log-format=json --requestheader-allowed-names=kube-apiserver-proxy-client --cloud-provider= --etcd-prefix=/registry --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --allow-privileged=true --service-account-lookup=true --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-policy-file=/etc/kubernetes/audit-policy.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --storage-backend=etcd3 --anonymous-auth=false --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --advertise-address=192.168.1.225 --audit-log-maxage=30 --etcd-servers=https://192.168.1.225:2379 --runtime-config=policy/v1beta1/podsecuritypolicy=true --bind-address=0.0.0.0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --insecure-port=0 --requestheader-group-headers=X-Remote-Group --secure-port=6443 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --requestheader-extra-headers-prefix=X-Remote-Extra- --profiling=false --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User - -``` -### 1.2.18 Ensure that the --insecure-bind-address argument is not set (Automated) - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the master node and remove the --insecure-bind-address parameter. - - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--insecure-bind-address' is not present -``` - -**Returned Value**: - -```console -root 4643 4626 22 16:15 ? 00:00:46 kube-apiserver --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authorization-mode=Node,RBAC --audit-log-maxsize=100 --audit-log-format=json --requestheader-allowed-names=kube-apiserver-proxy-client --cloud-provider= --etcd-prefix=/registry --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --allow-privileged=true --service-account-lookup=true --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-policy-file=/etc/kubernetes/audit-policy.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --storage-backend=etcd3 --anonymous-auth=false --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --advertise-address=192.168.1.225 --audit-log-maxage=30 --etcd-servers=https://192.168.1.225:2379 --runtime-config=policy/v1beta1/podsecuritypolicy=true --bind-address=0.0.0.0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --insecure-port=0 --requestheader-group-headers=X-Remote-Group --secure-port=6443 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --requestheader-extra-headers-prefix=X-Remote-Extra- --profiling=false --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User - -``` -### 1.2.19 Ensure that the --insecure-port argument is set to 0 (Automated) - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the master node and set the below parameter. ---insecure-port=0 - - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'0' is equal to '0' -``` - -**Returned Value**: - -```console -root 4643 4626 22 16:15 ? 00:00:46 kube-apiserver --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authorization-mode=Node,RBAC --audit-log-maxsize=100 --audit-log-format=json --requestheader-allowed-names=kube-apiserver-proxy-client --cloud-provider= --etcd-prefix=/registry --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --allow-privileged=true --service-account-lookup=true --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-policy-file=/etc/kubernetes/audit-policy.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --storage-backend=etcd3 --anonymous-auth=false --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --advertise-address=192.168.1.225 --audit-log-maxage=30 --etcd-servers=https://192.168.1.225:2379 --runtime-config=policy/v1beta1/podsecuritypolicy=true --bind-address=0.0.0.0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --insecure-port=0 --requestheader-group-headers=X-Remote-Group --secure-port=6443 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --requestheader-extra-headers-prefix=X-Remote-Extra- --profiling=false --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User - -``` -### 1.2.20 Ensure that the --secure-port argument is not set to 0 (Automated) - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the master node and either remove the --secure-port parameter or -set it to a different (non-zero) desired port. - - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -6443 is greater than 0 OR '--secure-port' is not present -``` - -**Returned Value**: - -```console -root 4643 4626 22 16:15 ? 00:00:46 kube-apiserver --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authorization-mode=Node,RBAC --audit-log-maxsize=100 --audit-log-format=json --requestheader-allowed-names=kube-apiserver-proxy-client --cloud-provider= --etcd-prefix=/registry --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --allow-privileged=true --service-account-lookup=true --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-policy-file=/etc/kubernetes/audit-policy.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --storage-backend=etcd3 --anonymous-auth=false --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --advertise-address=192.168.1.225 --audit-log-maxage=30 --etcd-servers=https://192.168.1.225:2379 --runtime-config=policy/v1beta1/podsecuritypolicy=true --bind-address=0.0.0.0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --insecure-port=0 --requestheader-group-headers=X-Remote-Group --secure-port=6443 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --requestheader-extra-headers-prefix=X-Remote-Extra- --profiling=false --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User - -``` -### 1.2.21 Ensure that the --profiling argument is set to false (Automated) - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the master node and set the below parameter. ---profiling=false - - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'false' is equal to 'false' -``` - -**Returned Value**: - -```console -root 4643 4626 22 16:15 ? 00:00:46 kube-apiserver --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authorization-mode=Node,RBAC --audit-log-maxsize=100 --audit-log-format=json --requestheader-allowed-names=kube-apiserver-proxy-client --cloud-provider= --etcd-prefix=/registry --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --allow-privileged=true --service-account-lookup=true --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-policy-file=/etc/kubernetes/audit-policy.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --storage-backend=etcd3 --anonymous-auth=false --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --advertise-address=192.168.1.225 --audit-log-maxage=30 --etcd-servers=https://192.168.1.225:2379 --runtime-config=policy/v1beta1/podsecuritypolicy=true --bind-address=0.0.0.0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --insecure-port=0 --requestheader-group-headers=X-Remote-Group --secure-port=6443 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --requestheader-extra-headers-prefix=X-Remote-Extra- --profiling=false --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User - -``` -### 1.2.22 Ensure that the --audit-log-path argument is set (Automated) - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the master node and set the --audit-log-path parameter to a suitable path and -file where you would like audit logs to be written, for example: ---audit-log-path=/var/log/apiserver/audit.log - - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--audit-log-path' is present -``` - -**Returned Value**: - -```console -root 4643 4626 22 16:15 ? 00:00:46 kube-apiserver --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authorization-mode=Node,RBAC --audit-log-maxsize=100 --audit-log-format=json --requestheader-allowed-names=kube-apiserver-proxy-client --cloud-provider= --etcd-prefix=/registry --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --allow-privileged=true --service-account-lookup=true --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-policy-file=/etc/kubernetes/audit-policy.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --storage-backend=etcd3 --anonymous-auth=false --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --advertise-address=192.168.1.225 --audit-log-maxage=30 --etcd-servers=https://192.168.1.225:2379 --runtime-config=policy/v1beta1/podsecuritypolicy=true --bind-address=0.0.0.0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --insecure-port=0 --requestheader-group-headers=X-Remote-Group --secure-port=6443 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --requestheader-extra-headers-prefix=X-Remote-Extra- --profiling=false --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User - -``` -### 1.2.23 Ensure that the --audit-log-maxage argument is set to 30 or as appropriate (Automated) - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the master node and set the --audit-log-maxage parameter to 30 or as an appropriate number of days: ---audit-log-maxage=30 - - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -30 is greater or equal to 30 -``` - -**Returned Value**: - -```console -root 4643 4626 22 16:15 ? 00:00:46 kube-apiserver --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authorization-mode=Node,RBAC --audit-log-maxsize=100 --audit-log-format=json --requestheader-allowed-names=kube-apiserver-proxy-client --cloud-provider= --etcd-prefix=/registry --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --allow-privileged=true --service-account-lookup=true --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-policy-file=/etc/kubernetes/audit-policy.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --storage-backend=etcd3 --anonymous-auth=false --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --advertise-address=192.168.1.225 --audit-log-maxage=30 --etcd-servers=https://192.168.1.225:2379 --runtime-config=policy/v1beta1/podsecuritypolicy=true --bind-address=0.0.0.0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --insecure-port=0 --requestheader-group-headers=X-Remote-Group --secure-port=6443 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --requestheader-extra-headers-prefix=X-Remote-Extra- --profiling=false --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User - -``` -### 1.2.24 Ensure that the --audit-log-maxbackup argument is set to 10 or as appropriate (Automated) - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the master node and set the --audit-log-maxbackup parameter to 10 or to an appropriate -value. ---audit-log-maxbackup=10 - - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -10 is greater or equal to 10 -``` - -**Returned Value**: - -```console -root 4643 4626 22 16:15 ? 00:00:46 kube-apiserver --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authorization-mode=Node,RBAC --audit-log-maxsize=100 --audit-log-format=json --requestheader-allowed-names=kube-apiserver-proxy-client --cloud-provider= --etcd-prefix=/registry --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --allow-privileged=true --service-account-lookup=true --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-policy-file=/etc/kubernetes/audit-policy.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --storage-backend=etcd3 --anonymous-auth=false --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --advertise-address=192.168.1.225 --audit-log-maxage=30 --etcd-servers=https://192.168.1.225:2379 --runtime-config=policy/v1beta1/podsecuritypolicy=true --bind-address=0.0.0.0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --insecure-port=0 --requestheader-group-headers=X-Remote-Group --secure-port=6443 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --requestheader-extra-headers-prefix=X-Remote-Extra- --profiling=false --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User - -``` -### 1.2.25 Ensure that the --audit-log-maxsize argument is set to 100 or as appropriate (Automated) - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the master node and set the --audit-log-maxsize parameter to an appropriate size in MB. -For example, to set it as 100 MB: ---audit-log-maxsize=100 - - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -100 is greater or equal to 100 -``` - -**Returned Value**: - -```console -root 4643 4626 22 16:15 ? 00:00:46 kube-apiserver --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authorization-mode=Node,RBAC --audit-log-maxsize=100 --audit-log-format=json --requestheader-allowed-names=kube-apiserver-proxy-client --cloud-provider= --etcd-prefix=/registry --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --allow-privileged=true --service-account-lookup=true --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-policy-file=/etc/kubernetes/audit-policy.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --storage-backend=etcd3 --anonymous-auth=false --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --advertise-address=192.168.1.225 --audit-log-maxage=30 --etcd-servers=https://192.168.1.225:2379 --runtime-config=policy/v1beta1/podsecuritypolicy=true --bind-address=0.0.0.0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --insecure-port=0 --requestheader-group-headers=X-Remote-Group --secure-port=6443 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --requestheader-extra-headers-prefix=X-Remote-Extra- --profiling=false --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User - -``` -### 1.2.26 Ensure that the --request-timeout argument is set as appropriate (Automated) - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -and set the below parameter as appropriate and if needed. -For example, ---request-timeout=300s - - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--request-timeout' is not present OR '--request-timeout' is not present -``` - -**Returned Value**: - -```console -root 4643 4626 22 16:15 ? 00:00:46 kube-apiserver --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authorization-mode=Node,RBAC --audit-log-maxsize=100 --audit-log-format=json --requestheader-allowed-names=kube-apiserver-proxy-client --cloud-provider= --etcd-prefix=/registry --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --allow-privileged=true --service-account-lookup=true --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-policy-file=/etc/kubernetes/audit-policy.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --storage-backend=etcd3 --anonymous-auth=false --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --advertise-address=192.168.1.225 --audit-log-maxage=30 --etcd-servers=https://192.168.1.225:2379 --runtime-config=policy/v1beta1/podsecuritypolicy=true --bind-address=0.0.0.0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --insecure-port=0 --requestheader-group-headers=X-Remote-Group --secure-port=6443 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --requestheader-extra-headers-prefix=X-Remote-Extra- --profiling=false --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User - -``` -### 1.2.27 Ensure that the --service-account-lookup argument is set to true (Automated) - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the master node and set the below parameter. ---service-account-lookup=true -Alternatively, you can delete the --service-account-lookup parameter from this file so -that the default takes effect. - - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--service-account-lookup' is not present OR 'true' is equal to 'true' -``` - -**Returned Value**: - -```console -root 4643 4626 22 16:15 ? 00:00:46 kube-apiserver --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authorization-mode=Node,RBAC --audit-log-maxsize=100 --audit-log-format=json --requestheader-allowed-names=kube-apiserver-proxy-client --cloud-provider= --etcd-prefix=/registry --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --allow-privileged=true --service-account-lookup=true --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-policy-file=/etc/kubernetes/audit-policy.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --storage-backend=etcd3 --anonymous-auth=false --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --advertise-address=192.168.1.225 --audit-log-maxage=30 --etcd-servers=https://192.168.1.225:2379 --runtime-config=policy/v1beta1/podsecuritypolicy=true --bind-address=0.0.0.0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --insecure-port=0 --requestheader-group-headers=X-Remote-Group --secure-port=6443 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --requestheader-extra-headers-prefix=X-Remote-Extra- --profiling=false --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User - -``` -### 1.2.28 Ensure that the --service-account-key-file argument is set as appropriate (Automated) - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the master node and set the --service-account-key-file parameter -to the public key file for service accounts: ---service-account-key-file= - - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--service-account-key-file' is present -``` - -**Returned Value**: - -```console -root 4643 4626 22 16:15 ? 00:00:46 kube-apiserver --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authorization-mode=Node,RBAC --audit-log-maxsize=100 --audit-log-format=json --requestheader-allowed-names=kube-apiserver-proxy-client --cloud-provider= --etcd-prefix=/registry --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --allow-privileged=true --service-account-lookup=true --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-policy-file=/etc/kubernetes/audit-policy.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --storage-backend=etcd3 --anonymous-auth=false --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --advertise-address=192.168.1.225 --audit-log-maxage=30 --etcd-servers=https://192.168.1.225:2379 --runtime-config=policy/v1beta1/podsecuritypolicy=true --bind-address=0.0.0.0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --insecure-port=0 --requestheader-group-headers=X-Remote-Group --secure-port=6443 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --requestheader-extra-headers-prefix=X-Remote-Extra- --profiling=false --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User - -``` -### 1.2.29 Ensure that the --etcd-certfile and --etcd-keyfile arguments are set as appropriate (Automated) - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection between the apiserver and etcd. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the master node and set the etcd certificate and key file parameters. ---etcd-certfile= ---etcd-keyfile= - - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--etcd-certfile' is present AND '--etcd-keyfile' is present -``` - -**Returned Value**: - -```console -root 4643 4626 22 16:15 ? 00:00:46 kube-apiserver --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authorization-mode=Node,RBAC --audit-log-maxsize=100 --audit-log-format=json --requestheader-allowed-names=kube-apiserver-proxy-client --cloud-provider= --etcd-prefix=/registry --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --allow-privileged=true --service-account-lookup=true --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-policy-file=/etc/kubernetes/audit-policy.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --storage-backend=etcd3 --anonymous-auth=false --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --advertise-address=192.168.1.225 --audit-log-maxage=30 --etcd-servers=https://192.168.1.225:2379 --runtime-config=policy/v1beta1/podsecuritypolicy=true --bind-address=0.0.0.0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --insecure-port=0 --requestheader-group-headers=X-Remote-Group --secure-port=6443 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --requestheader-extra-headers-prefix=X-Remote-Extra- --profiling=false --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User - -``` -### 1.2.30 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Automated) - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection on the apiserver. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the master node and set the TLS certificate and private key file parameters. ---tls-cert-file= ---tls-private-key-file= - - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--tls-cert-file' is present AND '--tls-private-key-file' is present -``` - -**Returned Value**: - -```console -root 4643 4626 22 16:15 ? 00:00:46 kube-apiserver --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authorization-mode=Node,RBAC --audit-log-maxsize=100 --audit-log-format=json --requestheader-allowed-names=kube-apiserver-proxy-client --cloud-provider= --etcd-prefix=/registry --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --allow-privileged=true --service-account-lookup=true --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-policy-file=/etc/kubernetes/audit-policy.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --storage-backend=etcd3 --anonymous-auth=false --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --advertise-address=192.168.1.225 --audit-log-maxage=30 --etcd-servers=https://192.168.1.225:2379 --runtime-config=policy/v1beta1/podsecuritypolicy=true --bind-address=0.0.0.0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --insecure-port=0 --requestheader-group-headers=X-Remote-Group --secure-port=6443 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --requestheader-extra-headers-prefix=X-Remote-Extra- --profiling=false --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User - -``` -### 1.2.31 Ensure that the --client-ca-file argument is set as appropriate (Automated) - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection on the apiserver. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the master node and set the client certificate authority file. ---client-ca-file= - - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--client-ca-file' is present -``` - -**Returned Value**: - -```console -root 4643 4626 22 16:15 ? 00:00:46 kube-apiserver --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authorization-mode=Node,RBAC --audit-log-maxsize=100 --audit-log-format=json --requestheader-allowed-names=kube-apiserver-proxy-client --cloud-provider= --etcd-prefix=/registry --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --allow-privileged=true --service-account-lookup=true --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-policy-file=/etc/kubernetes/audit-policy.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --storage-backend=etcd3 --anonymous-auth=false --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --advertise-address=192.168.1.225 --audit-log-maxage=30 --etcd-servers=https://192.168.1.225:2379 --runtime-config=policy/v1beta1/podsecuritypolicy=true --bind-address=0.0.0.0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --insecure-port=0 --requestheader-group-headers=X-Remote-Group --secure-port=6443 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --requestheader-extra-headers-prefix=X-Remote-Extra- --profiling=false --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User - -``` -### 1.2.32 Ensure that the --etcd-cafile argument is set as appropriate (Automated) - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection between the apiserver and etcd. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the master node and set the etcd certificate authority file parameter. ---etcd-cafile= - - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--etcd-cafile' is present -``` - -**Returned Value**: - -```console -root 4643 4626 22 16:15 ? 00:00:46 kube-apiserver --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authorization-mode=Node,RBAC --audit-log-maxsize=100 --audit-log-format=json --requestheader-allowed-names=kube-apiserver-proxy-client --cloud-provider= --etcd-prefix=/registry --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --allow-privileged=true --service-account-lookup=true --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-policy-file=/etc/kubernetes/audit-policy.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --storage-backend=etcd3 --anonymous-auth=false --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --advertise-address=192.168.1.225 --audit-log-maxage=30 --etcd-servers=https://192.168.1.225:2379 --runtime-config=policy/v1beta1/podsecuritypolicy=true --bind-address=0.0.0.0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --insecure-port=0 --requestheader-group-headers=X-Remote-Group --secure-port=6443 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --requestheader-extra-headers-prefix=X-Remote-Extra- --profiling=false --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User - -``` -### 1.2.33 Ensure that the --encryption-provider-config argument is set as appropriate (Automated) - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and configure a EncryptionConfig file. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the master node and set the --encryption-provider-config parameter to the path of that file: --encryption-provider-config= - - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--encryption-provider-config' is present -``` - -**Returned Value**: - -```console -root 4643 4626 22 16:15 ? 00:00:46 kube-apiserver --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authorization-mode=Node,RBAC --audit-log-maxsize=100 --audit-log-format=json --requestheader-allowed-names=kube-apiserver-proxy-client --cloud-provider= --etcd-prefix=/registry --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --allow-privileged=true --service-account-lookup=true --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-policy-file=/etc/kubernetes/audit-policy.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --storage-backend=etcd3 --anonymous-auth=false --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --advertise-address=192.168.1.225 --audit-log-maxage=30 --etcd-servers=https://192.168.1.225:2379 --runtime-config=policy/v1beta1/podsecuritypolicy=true --bind-address=0.0.0.0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --insecure-port=0 --requestheader-group-headers=X-Remote-Group --secure-port=6443 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --requestheader-extra-headers-prefix=X-Remote-Extra- --profiling=false --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User - -``` -### 1.2.34 Ensure that encryption providers are appropriately configured (Automated) - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and configure a EncryptionConfig file. -In this file, choose aescbc, kms or secretbox as the encryption provider. - - -**Audit:** - -```bash -check_encryption_provider_config.sh aescbc kms secretbox -``` - -**Expected Result**: - -```console -'true' is equal to 'true' -``` - -**Audit Script:** -```bash -#!/usr/bin/env bash - -# This script is used to check the encrption provider config is set to aesbc -# -# outputs: -# true/false - -# TODO: Figure out the file location from the kube-apiserver commandline args -ENCRYPTION_CONFIG_FILE="/node/etc/kubernetes/ssl/encryption.yaml" - -if [[ ! -f "${ENCRYPTION_CONFIG_FILE}" ]]; then - echo "false" - exit -fi - -for provider in "$@" -do - if grep "$provider" "${ENCRYPTION_CONFIG_FILE}"; then - echo "true" - exit - fi -done - -echo "false" -exit - -``` -**Returned Value**: - -```console - - aescbc: -true - -``` -### 1.2.35 Ensure that the API Server only makes use of Strong Cryptographic Ciphers (Automated) - -**Result:** warn - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the master node and set the below parameter. ---tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM -_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM -_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM -_SHA384 - - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - - -## 1.3 Controller Manager -### 1.3.1 Ensure that the --terminated-pod-gc-threshold argument is set as appropriate (Automated) - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the master node and set the --terminated-pod-gc-threshold to an appropriate threshold, -for example: ---terminated-pod-gc-threshold=10 - - -**Audit:** - -```bash -/bin/ps -ef | grep kube-controller-manager | grep -v grep -``` - -**Expected Result**: - -```console -'--terminated-pod-gc-threshold' is present -``` - -**Returned Value**: - -```console -root 4788 4773 4 16:16 ? 00:00:09 kube-controller-manager --configure-cloud-routes=false --cloud-provider= --service-cluster-ip-range=10.43.0.0/16 --v=2 --pod-eviction-timeout=5m0s --leader-elect=true --cluster-cidr=10.42.0.0/16 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --profiling=false --node-monitor-grace-period=40s --service-account-private-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --address=0.0.0.0 --allow-untagged-cloud=true --root-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allocate-node-cidrs=true --enable-hostpath-provisioner=false --terminated-pod-gc-threshold=1000 --feature-gates=RotateKubeletServerCertificate=true --use-service-account-credentials=true - -``` -### 1.3.2 Ensure that the --profiling argument is set to false (Automated) - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the master node and set the below parameter. ---profiling=false - - -**Audit:** - -```bash -/bin/ps -ef | grep kube-controller-manager | grep -v grep -``` - -**Expected Result**: - -```console -'false' is equal to 'false' -``` - -**Returned Value**: - -```console -root 4788 4773 4 16:16 ? 00:00:09 kube-controller-manager --configure-cloud-routes=false --cloud-provider= --service-cluster-ip-range=10.43.0.0/16 --v=2 --pod-eviction-timeout=5m0s --leader-elect=true --cluster-cidr=10.42.0.0/16 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --profiling=false --node-monitor-grace-period=40s --service-account-private-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --address=0.0.0.0 --allow-untagged-cloud=true --root-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allocate-node-cidrs=true --enable-hostpath-provisioner=false --terminated-pod-gc-threshold=1000 --feature-gates=RotateKubeletServerCertificate=true --use-service-account-credentials=true - -``` -### 1.3.3 Ensure that the --use-service-account-credentials argument is set to true (Automated) - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the master node to set the below parameter. ---use-service-account-credentials=true - - -**Audit:** - -```bash -/bin/ps -ef | grep kube-controller-manager | grep -v grep -``` - -**Expected Result**: - -```console -'true' is not equal to 'false' -``` - -**Returned Value**: - -```console -root 4788 4773 4 16:16 ? 00:00:09 kube-controller-manager --configure-cloud-routes=false --cloud-provider= --service-cluster-ip-range=10.43.0.0/16 --v=2 --pod-eviction-timeout=5m0s --leader-elect=true --cluster-cidr=10.42.0.0/16 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --profiling=false --node-monitor-grace-period=40s --service-account-private-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --address=0.0.0.0 --allow-untagged-cloud=true --root-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allocate-node-cidrs=true --enable-hostpath-provisioner=false --terminated-pod-gc-threshold=1000 --feature-gates=RotateKubeletServerCertificate=true --use-service-account-credentials=true - -``` -### 1.3.4 Ensure that the --service-account-private-key-file argument is set as appropriate (Automated) - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the master node and set the --service-account-private-key-file parameter -to the private key file for service accounts. ---service-account-private-key-file= - - -**Audit:** - -```bash -/bin/ps -ef | grep kube-controller-manager | grep -v grep -``` - -**Expected Result**: - -```console -'--service-account-private-key-file' is present -``` - -**Returned Value**: - -```console -root 4788 4773 4 16:16 ? 00:00:09 kube-controller-manager --configure-cloud-routes=false --cloud-provider= --service-cluster-ip-range=10.43.0.0/16 --v=2 --pod-eviction-timeout=5m0s --leader-elect=true --cluster-cidr=10.42.0.0/16 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --profiling=false --node-monitor-grace-period=40s --service-account-private-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --address=0.0.0.0 --allow-untagged-cloud=true --root-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allocate-node-cidrs=true --enable-hostpath-provisioner=false --terminated-pod-gc-threshold=1000 --feature-gates=RotateKubeletServerCertificate=true --use-service-account-credentials=true - -``` -### 1.3.5 Ensure that the --root-ca-file argument is set as appropriate (Automated) - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the master node and set the --root-ca-file parameter to the certificate bundle file`. ---root-ca-file= - - -**Audit:** - -```bash -/bin/ps -ef | grep kube-controller-manager | grep -v grep -``` - -**Expected Result**: - -```console -'--root-ca-file' is present -``` - -**Returned Value**: - -```console -root 4788 4773 4 16:16 ? 00:00:09 kube-controller-manager --configure-cloud-routes=false --cloud-provider= --service-cluster-ip-range=10.43.0.0/16 --v=2 --pod-eviction-timeout=5m0s --leader-elect=true --cluster-cidr=10.42.0.0/16 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --profiling=false --node-monitor-grace-period=40s --service-account-private-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --address=0.0.0.0 --allow-untagged-cloud=true --root-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allocate-node-cidrs=true --enable-hostpath-provisioner=false --terminated-pod-gc-threshold=1000 --feature-gates=RotateKubeletServerCertificate=true --use-service-account-credentials=true - -``` -### 1.3.6 Ensure that the RotateKubeletServerCertificate argument is set to true (Automated) - -**Result:** notApplicable - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the master node and set the --feature-gates parameter to include RotateKubeletServerCertificate=true. ---feature-gates=RotateKubeletServerCertificate=true - -Cluster provisioned by RKE handles certificate rotation directly through RKE. - - -**Audit:** - -```bash -/bin/ps -ef | grep kube-controller-manager | grep -v grep -``` - - -### 1.3.7 Ensure that the --bind-address argument is set to 127.0.0.1 (Automated) - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the master node and ensure the correct value for the --bind-address parameter - - -**Audit:** - -```bash -/bin/ps -ef | grep kube-controller-manager | grep -v grep -``` - -**Expected Result**: - -```console -'--bind-address' is not present OR '--bind-address' is not present -``` - -**Returned Value**: - -```console -root 4788 4773 4 16:16 ? 00:00:09 kube-controller-manager --configure-cloud-routes=false --cloud-provider= --service-cluster-ip-range=10.43.0.0/16 --v=2 --pod-eviction-timeout=5m0s --leader-elect=true --cluster-cidr=10.42.0.0/16 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --profiling=false --node-monitor-grace-period=40s --service-account-private-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --address=0.0.0.0 --allow-untagged-cloud=true --root-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allocate-node-cidrs=true --enable-hostpath-provisioner=false --terminated-pod-gc-threshold=1000 --feature-gates=RotateKubeletServerCertificate=true --use-service-account-credentials=true - -``` -## 1.4 Scheduler -### 1.4.1 Ensure that the --profiling argument is set to false (Automated) - -**Result:** pass - -**Remediation:** -Edit the Scheduler pod specification file /etc/kubernetes/manifests/kube-scheduler.yaml file -on the master node and set the below parameter. ---profiling=false - - -**Audit:** - -```bash -/bin/ps -ef | grep kube-scheduler | grep -v grep -``` - -**Expected Result**: - -```console -'false' is equal to 'false' -``` - -**Returned Value**: - -```console -root 4947 4930 1 16:16 ? 00:00:02 kube-scheduler --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-scheduler.yaml --leader-elect=true --profiling=false --v=2 --address=0.0.0.0 - -``` -### 1.4.2 Ensure that the --bind-address argument is set to 127.0.0.1 (Automated) - -**Result:** pass - -**Remediation:** -Edit the Scheduler pod specification file /etc/kubernetes/manifests/kube-scheduler.yaml -on the master node and ensure the correct value for the --bind-address parameter - - -**Audit:** - -```bash -/bin/ps -ef | grep kube-scheduler | grep -v grep -``` - -**Expected Result**: - -```console -'--bind-address' is not present OR '--bind-address' is not present -``` - -**Returned Value**: - -```console -root 4947 4930 1 16:16 ? 00:00:02 kube-scheduler --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-scheduler.yaml --leader-elect=true --profiling=false --v=2 --address=0.0.0.0 - -``` -## 2 Etcd Node Configuration Files -### 2.1 Ensure that the --cert-file and --key-file arguments are set as appropriate (Automated) - -**Result:** pass - -**Remediation:** -Follow the etcd service documentation and configure TLS encryption. -Then, edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml -on the master node and set the below parameters. ---cert-file= ---key-file= - - -**Audit:** - -```bash -/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep -``` - -**Expected Result**: - -```console -'--cert-file' is present AND '--key-file' is present -``` - -**Returned Value**: - -```console -etcd 4318 4301 6 16:15 ? 00:00:14 /usr/local/bin/etcd --listen-peer-urls=https://0.0.0.0:2380 --cert-file=/etc/kubernetes/ssl/kube-etcd-192-168-1-225.pem --advertise-client-urls=https://192.168.1.225:2379,https://192.168.1.225:4001 --election-timeout=5000 --data-dir=/var/lib/rancher/etcd/ --peer-cert-file=/etc/kubernetes/ssl/kube-etcd-192-168-1-225.pem --enable-v2=true --initial-cluster=etcd-cis-aio-0=https://192.168.1.225:2380 --trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --peer-trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --client-cert-auth=true --heartbeat-interval=500 --initial-cluster-token=etcd-cluster-1 --name=etcd-cis-aio-0 --listen-client-urls=https://0.0.0.0:2379 --peer-key-file=/etc/kubernetes/ssl/kube-etcd-192-168-1-225-key.pem --peer-client-cert-auth=true --initial-advertise-peer-urls=https://192.168.1.225:2380 --initial-cluster-state=new --key-file=/etc/kubernetes/ssl/kube-etcd-192-168-1-225-key.pem -root 4366 4349 0 16:15 ? 00:00:00 /opt/rke-tools/rke-etcd-backup etcd-backup save --cacert /etc/kubernetes/ssl/kube-ca.pem --cert /etc/kubernetes/ssl/kube-node.pem --key /etc/kubernetes/ssl/kube-node-key.pem --name etcd-rolling-snapshots --endpoints=192.168.1.225:2379 --retention=72h --creation=12h -root 4643 4626 23 16:15 ? 00:00:46 kube-apiserver --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authorization-mode=Node,RBAC --audit-log-maxsize=100 --audit-log-format=json --requestheader-allowed-names=kube-apiserver-proxy-client --cloud-provider= --etcd-prefix=/registry --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --allow-privileged=true --service-account-lookup=true --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-policy-file=/etc/kubernetes/audit-policy.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --storage-backend=etcd3 --anonymous-auth=false --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --advertise-address=192.168.1.225 --audit-log-maxage=30 --etcd-servers=https://192.168.1.225:2379 --runtime-config=policy/v1beta1/podsecuritypolicy=true --bind-address=0.0.0.0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --insecure-port=0 --requestheader-group-headers=X-Remote-Group --secure-port=6443 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --requestheader-extra-headers-prefix=X-Remote-Extra- --profiling=false --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User -root 14998 14985 0 16:19 ? 00:00:00 kube-bench run --targets etcd --scored --nosummary --noremediations --v=5 --config-dir=/etc/kube-bench/cfg --benchmark rke-cis-1.6-hardened --json --log_dir /tmp/results/logs --outputfile /tmp/results/etcd.json - -``` -### 2.2 Ensure that the --client-cert-auth argument is set to true (Automated) - -**Result:** pass - -**Remediation:** -Edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the master -node and set the below parameter. ---client-cert-auth="true" - - -**Audit:** - -```bash -/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep -``` - -**Expected Result**: - -```console -'--client-cert-auth' is present OR 'true' is equal to 'true' -``` - -**Returned Value**: - -```console -etcd 4318 4301 6 16:15 ? 00:00:14 /usr/local/bin/etcd --listen-peer-urls=https://0.0.0.0:2380 --cert-file=/etc/kubernetes/ssl/kube-etcd-192-168-1-225.pem --advertise-client-urls=https://192.168.1.225:2379,https://192.168.1.225:4001 --election-timeout=5000 --data-dir=/var/lib/rancher/etcd/ --peer-cert-file=/etc/kubernetes/ssl/kube-etcd-192-168-1-225.pem --enable-v2=true --initial-cluster=etcd-cis-aio-0=https://192.168.1.225:2380 --trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --peer-trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --client-cert-auth=true --heartbeat-interval=500 --initial-cluster-token=etcd-cluster-1 --name=etcd-cis-aio-0 --listen-client-urls=https://0.0.0.0:2379 --peer-key-file=/etc/kubernetes/ssl/kube-etcd-192-168-1-225-key.pem --peer-client-cert-auth=true --initial-advertise-peer-urls=https://192.168.1.225:2380 --initial-cluster-state=new --key-file=/etc/kubernetes/ssl/kube-etcd-192-168-1-225-key.pem -root 4366 4349 0 16:15 ? 00:00:00 /opt/rke-tools/rke-etcd-backup etcd-backup save --cacert /etc/kubernetes/ssl/kube-ca.pem --cert /etc/kubernetes/ssl/kube-node.pem --key /etc/kubernetes/ssl/kube-node-key.pem --name etcd-rolling-snapshots --endpoints=192.168.1.225:2379 --retention=72h --creation=12h -root 4643 4626 23 16:15 ? 00:00:46 kube-apiserver --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authorization-mode=Node,RBAC --audit-log-maxsize=100 --audit-log-format=json --requestheader-allowed-names=kube-apiserver-proxy-client --cloud-provider= --etcd-prefix=/registry --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --allow-privileged=true --service-account-lookup=true --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-policy-file=/etc/kubernetes/audit-policy.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --storage-backend=etcd3 --anonymous-auth=false --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --advertise-address=192.168.1.225 --audit-log-maxage=30 --etcd-servers=https://192.168.1.225:2379 --runtime-config=policy/v1beta1/podsecuritypolicy=true --bind-address=0.0.0.0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --insecure-port=0 --requestheader-group-headers=X-Remote-Group --secure-port=6443 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --requestheader-extra-headers-prefix=X-Remote-Extra- --profiling=false --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User -root 14998 14985 0 16:19 ? 00:00:00 kube-bench run --targets etcd --scored --nosummary --noremediations --v=5 --config-dir=/etc/kube-bench/cfg --benchmark rke-cis-1.6-hardened --json --log_dir /tmp/results/logs --outputfile /tmp/results/etcd.json - -``` -### 2.3 Ensure that the --auto-tls argument is not set to true (Automated) - -**Result:** pass - -**Remediation:** -Edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the master -node and either remove the --auto-tls parameter or set it to false. - --auto-tls=false - - -**Audit:** - -```bash -/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep -``` - -**Expected Result**: - -```console -'--auto-tls' is not present OR '--auto-tls' is not present -``` - -**Returned Value**: - -```console -etcd 4318 4301 6 16:15 ? 00:00:14 /usr/local/bin/etcd --listen-peer-urls=https://0.0.0.0:2380 --cert-file=/etc/kubernetes/ssl/kube-etcd-192-168-1-225.pem --advertise-client-urls=https://192.168.1.225:2379,https://192.168.1.225:4001 --election-timeout=5000 --data-dir=/var/lib/rancher/etcd/ --peer-cert-file=/etc/kubernetes/ssl/kube-etcd-192-168-1-225.pem --enable-v2=true --initial-cluster=etcd-cis-aio-0=https://192.168.1.225:2380 --trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --peer-trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --client-cert-auth=true --heartbeat-interval=500 --initial-cluster-token=etcd-cluster-1 --name=etcd-cis-aio-0 --listen-client-urls=https://0.0.0.0:2379 --peer-key-file=/etc/kubernetes/ssl/kube-etcd-192-168-1-225-key.pem --peer-client-cert-auth=true --initial-advertise-peer-urls=https://192.168.1.225:2380 --initial-cluster-state=new --key-file=/etc/kubernetes/ssl/kube-etcd-192-168-1-225-key.pem -root 4366 4349 0 16:15 ? 00:00:00 /opt/rke-tools/rke-etcd-backup etcd-backup save --cacert /etc/kubernetes/ssl/kube-ca.pem --cert /etc/kubernetes/ssl/kube-node.pem --key /etc/kubernetes/ssl/kube-node-key.pem --name etcd-rolling-snapshots --endpoints=192.168.1.225:2379 --retention=72h --creation=12h -root 4643 4626 23 16:15 ? 00:00:46 kube-apiserver --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authorization-mode=Node,RBAC --audit-log-maxsize=100 --audit-log-format=json --requestheader-allowed-names=kube-apiserver-proxy-client --cloud-provider= --etcd-prefix=/registry --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --allow-privileged=true --service-account-lookup=true --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-policy-file=/etc/kubernetes/audit-policy.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --storage-backend=etcd3 --anonymous-auth=false --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --advertise-address=192.168.1.225 --audit-log-maxage=30 --etcd-servers=https://192.168.1.225:2379 --runtime-config=policy/v1beta1/podsecuritypolicy=true --bind-address=0.0.0.0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --insecure-port=0 --requestheader-group-headers=X-Remote-Group --secure-port=6443 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --requestheader-extra-headers-prefix=X-Remote-Extra- --profiling=false --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User -root 14998 14985 0 16:19 ? 00:00:00 kube-bench run --targets etcd --scored --nosummary --noremediations --v=5 --config-dir=/etc/kube-bench/cfg --benchmark rke-cis-1.6-hardened --json --log_dir /tmp/results/logs --outputfile /tmp/results/etcd.json - -``` -### 2.4 Ensure that the --peer-cert-file and --peer-key-file arguments are set as appropriate (Automated) - -**Result:** pass - -**Remediation:** -Follow the etcd service documentation and configure peer TLS encryption as appropriate -for your etcd cluster. -Then, edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the -master node and set the below parameters. ---peer-client-file= ---peer-key-file= - - -**Audit:** - -```bash -/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep -``` - -**Expected Result**: - -```console -'--peer-cert-file' is present AND '--peer-key-file' is present -``` - -**Returned Value**: - -```console -etcd 4318 4301 6 16:15 ? 00:00:14 /usr/local/bin/etcd --listen-peer-urls=https://0.0.0.0:2380 --cert-file=/etc/kubernetes/ssl/kube-etcd-192-168-1-225.pem --advertise-client-urls=https://192.168.1.225:2379,https://192.168.1.225:4001 --election-timeout=5000 --data-dir=/var/lib/rancher/etcd/ --peer-cert-file=/etc/kubernetes/ssl/kube-etcd-192-168-1-225.pem --enable-v2=true --initial-cluster=etcd-cis-aio-0=https://192.168.1.225:2380 --trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --peer-trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --client-cert-auth=true --heartbeat-interval=500 --initial-cluster-token=etcd-cluster-1 --name=etcd-cis-aio-0 --listen-client-urls=https://0.0.0.0:2379 --peer-key-file=/etc/kubernetes/ssl/kube-etcd-192-168-1-225-key.pem --peer-client-cert-auth=true --initial-advertise-peer-urls=https://192.168.1.225:2380 --initial-cluster-state=new --key-file=/etc/kubernetes/ssl/kube-etcd-192-168-1-225-key.pem -root 4366 4349 0 16:15 ? 00:00:00 /opt/rke-tools/rke-etcd-backup etcd-backup save --cacert /etc/kubernetes/ssl/kube-ca.pem --cert /etc/kubernetes/ssl/kube-node.pem --key /etc/kubernetes/ssl/kube-node-key.pem --name etcd-rolling-snapshots --endpoints=192.168.1.225:2379 --retention=72h --creation=12h -root 4643 4626 23 16:15 ? 00:00:46 kube-apiserver --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authorization-mode=Node,RBAC --audit-log-maxsize=100 --audit-log-format=json --requestheader-allowed-names=kube-apiserver-proxy-client --cloud-provider= --etcd-prefix=/registry --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --allow-privileged=true --service-account-lookup=true --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-policy-file=/etc/kubernetes/audit-policy.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --storage-backend=etcd3 --anonymous-auth=false --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --advertise-address=192.168.1.225 --audit-log-maxage=30 --etcd-servers=https://192.168.1.225:2379 --runtime-config=policy/v1beta1/podsecuritypolicy=true --bind-address=0.0.0.0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --insecure-port=0 --requestheader-group-headers=X-Remote-Group --secure-port=6443 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --requestheader-extra-headers-prefix=X-Remote-Extra- --profiling=false --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User -root 14998 14985 0 16:19 ? 00:00:00 kube-bench run --targets etcd --scored --nosummary --noremediations --v=5 --config-dir=/etc/kube-bench/cfg --benchmark rke-cis-1.6-hardened --json --log_dir /tmp/results/logs --outputfile /tmp/results/etcd.json - -``` -### 2.5 Ensure that the --peer-client-cert-auth argument is set to true (Automated) - -**Result:** pass - -**Remediation:** -Edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the master -node and set the below parameter. ---peer-client-cert-auth=true - - -**Audit:** - -```bash -/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep -``` - -**Expected Result**: - -```console -'--peer-client-cert-auth' is present OR 'true' is equal to 'true' -``` - -**Returned Value**: - -```console -etcd 4318 4301 6 16:15 ? 00:00:14 /usr/local/bin/etcd --listen-peer-urls=https://0.0.0.0:2380 --cert-file=/etc/kubernetes/ssl/kube-etcd-192-168-1-225.pem --advertise-client-urls=https://192.168.1.225:2379,https://192.168.1.225:4001 --election-timeout=5000 --data-dir=/var/lib/rancher/etcd/ --peer-cert-file=/etc/kubernetes/ssl/kube-etcd-192-168-1-225.pem --enable-v2=true --initial-cluster=etcd-cis-aio-0=https://192.168.1.225:2380 --trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --peer-trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --client-cert-auth=true --heartbeat-interval=500 --initial-cluster-token=etcd-cluster-1 --name=etcd-cis-aio-0 --listen-client-urls=https://0.0.0.0:2379 --peer-key-file=/etc/kubernetes/ssl/kube-etcd-192-168-1-225-key.pem --peer-client-cert-auth=true --initial-advertise-peer-urls=https://192.168.1.225:2380 --initial-cluster-state=new --key-file=/etc/kubernetes/ssl/kube-etcd-192-168-1-225-key.pem -root 4366 4349 0 16:15 ? 00:00:00 /opt/rke-tools/rke-etcd-backup etcd-backup save --cacert /etc/kubernetes/ssl/kube-ca.pem --cert /etc/kubernetes/ssl/kube-node.pem --key /etc/kubernetes/ssl/kube-node-key.pem --name etcd-rolling-snapshots --endpoints=192.168.1.225:2379 --retention=72h --creation=12h -root 4643 4626 23 16:15 ? 00:00:46 kube-apiserver --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authorization-mode=Node,RBAC --audit-log-maxsize=100 --audit-log-format=json --requestheader-allowed-names=kube-apiserver-proxy-client --cloud-provider= --etcd-prefix=/registry --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --allow-privileged=true --service-account-lookup=true --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-policy-file=/etc/kubernetes/audit-policy.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --storage-backend=etcd3 --anonymous-auth=false --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --advertise-address=192.168.1.225 --audit-log-maxage=30 --etcd-servers=https://192.168.1.225:2379 --runtime-config=policy/v1beta1/podsecuritypolicy=true --bind-address=0.0.0.0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --insecure-port=0 --requestheader-group-headers=X-Remote-Group --secure-port=6443 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --requestheader-extra-headers-prefix=X-Remote-Extra- --profiling=false --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User -root 14998 14985 0 16:19 ? 00:00:00 kube-bench run --targets etcd --scored --nosummary --noremediations --v=5 --config-dir=/etc/kube-bench/cfg --benchmark rke-cis-1.6-hardened --json --log_dir /tmp/results/logs --outputfile /tmp/results/etcd.json - -``` -### 2.6 Ensure that the --peer-auto-tls argument is not set to true (Automated) - -**Result:** pass - -**Remediation:** -Edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the master -node and either remove the --peer-auto-tls parameter or set it to false. ---peer-auto-tls=false - - -**Audit:** - -```bash -/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep -``` - -**Expected Result**: - -```console -'--peer-auto-tls' is not present OR '--peer-auto-tls' is present -``` - -**Returned Value**: - -```console -etcd 4318 4301 6 16:15 ? 00:00:14 /usr/local/bin/etcd --listen-peer-urls=https://0.0.0.0:2380 --cert-file=/etc/kubernetes/ssl/kube-etcd-192-168-1-225.pem --advertise-client-urls=https://192.168.1.225:2379,https://192.168.1.225:4001 --election-timeout=5000 --data-dir=/var/lib/rancher/etcd/ --peer-cert-file=/etc/kubernetes/ssl/kube-etcd-192-168-1-225.pem --enable-v2=true --initial-cluster=etcd-cis-aio-0=https://192.168.1.225:2380 --trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --peer-trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --client-cert-auth=true --heartbeat-interval=500 --initial-cluster-token=etcd-cluster-1 --name=etcd-cis-aio-0 --listen-client-urls=https://0.0.0.0:2379 --peer-key-file=/etc/kubernetes/ssl/kube-etcd-192-168-1-225-key.pem --peer-client-cert-auth=true --initial-advertise-peer-urls=https://192.168.1.225:2380 --initial-cluster-state=new --key-file=/etc/kubernetes/ssl/kube-etcd-192-168-1-225-key.pem -root 4366 4349 0 16:15 ? 00:00:00 /opt/rke-tools/rke-etcd-backup etcd-backup save --cacert /etc/kubernetes/ssl/kube-ca.pem --cert /etc/kubernetes/ssl/kube-node.pem --key /etc/kubernetes/ssl/kube-node-key.pem --name etcd-rolling-snapshots --endpoints=192.168.1.225:2379 --retention=72h --creation=12h -root 4643 4626 23 16:15 ? 00:00:46 kube-apiserver --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authorization-mode=Node,RBAC --audit-log-maxsize=100 --audit-log-format=json --requestheader-allowed-names=kube-apiserver-proxy-client --cloud-provider= --etcd-prefix=/registry --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --allow-privileged=true --service-account-lookup=true --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-policy-file=/etc/kubernetes/audit-policy.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --storage-backend=etcd3 --anonymous-auth=false --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --advertise-address=192.168.1.225 --audit-log-maxage=30 --etcd-servers=https://192.168.1.225:2379 --runtime-config=policy/v1beta1/podsecuritypolicy=true --bind-address=0.0.0.0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --insecure-port=0 --requestheader-group-headers=X-Remote-Group --secure-port=6443 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --requestheader-extra-headers-prefix=X-Remote-Extra- --profiling=false --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User -root 14998 14985 0 16:19 ? 00:00:00 kube-bench run --targets etcd --scored --nosummary --noremediations --v=5 --config-dir=/etc/kube-bench/cfg --benchmark rke-cis-1.6-hardened --json --log_dir /tmp/results/logs --outputfile /tmp/results/etcd.json - -``` -### 2.7 Ensure that a unique Certificate Authority is used for etcd (Automated) - -**Result:** pass - -**Remediation:** -[Manual test] -Follow the etcd documentation and create a dedicated certificate authority setup for the -etcd service. -Then, edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the -master node and set the below parameter. ---trusted-ca-file= - - -**Audit:** - -```bash -/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep -``` - -**Expected Result**: - -```console -'--trusted-ca-file' is present -``` - -**Returned Value**: - -```console -etcd 4318 4301 6 16:15 ? 00:00:14 /usr/local/bin/etcd --listen-peer-urls=https://0.0.0.0:2380 --cert-file=/etc/kubernetes/ssl/kube-etcd-192-168-1-225.pem --advertise-client-urls=https://192.168.1.225:2379,https://192.168.1.225:4001 --election-timeout=5000 --data-dir=/var/lib/rancher/etcd/ --peer-cert-file=/etc/kubernetes/ssl/kube-etcd-192-168-1-225.pem --enable-v2=true --initial-cluster=etcd-cis-aio-0=https://192.168.1.225:2380 --trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --peer-trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --client-cert-auth=true --heartbeat-interval=500 --initial-cluster-token=etcd-cluster-1 --name=etcd-cis-aio-0 --listen-client-urls=https://0.0.0.0:2379 --peer-key-file=/etc/kubernetes/ssl/kube-etcd-192-168-1-225-key.pem --peer-client-cert-auth=true --initial-advertise-peer-urls=https://192.168.1.225:2380 --initial-cluster-state=new --key-file=/etc/kubernetes/ssl/kube-etcd-192-168-1-225-key.pem -root 4366 4349 0 16:15 ? 00:00:00 /opt/rke-tools/rke-etcd-backup etcd-backup save --cacert /etc/kubernetes/ssl/kube-ca.pem --cert /etc/kubernetes/ssl/kube-node.pem --key /etc/kubernetes/ssl/kube-node-key.pem --name etcd-rolling-snapshots --endpoints=192.168.1.225:2379 --retention=72h --creation=12h -root 4643 4626 23 16:15 ? 00:00:46 kube-apiserver --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authorization-mode=Node,RBAC --audit-log-maxsize=100 --audit-log-format=json --requestheader-allowed-names=kube-apiserver-proxy-client --cloud-provider= --etcd-prefix=/registry --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --allow-privileged=true --service-account-lookup=true --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-policy-file=/etc/kubernetes/audit-policy.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --storage-backend=etcd3 --anonymous-auth=false --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --advertise-address=192.168.1.225 --audit-log-maxage=30 --etcd-servers=https://192.168.1.225:2379 --runtime-config=policy/v1beta1/podsecuritypolicy=true --bind-address=0.0.0.0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --insecure-port=0 --requestheader-group-headers=X-Remote-Group --secure-port=6443 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --requestheader-extra-headers-prefix=X-Remote-Extra- --profiling=false --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User -root 14998 14985 0 16:19 ? 00:00:00 kube-bench run --targets etcd --scored --nosummary --noremediations --v=5 --config-dir=/etc/kube-bench/cfg --benchmark rke-cis-1.6-hardened --json --log_dir /tmp/results/logs --outputfile /tmp/results/etcd.json - -``` -## 3.1 Authentication and Authorization -### 3.1.1 Client certificate authentication should not be used for users (Manual) - -**Result:** warn - -**Remediation:** -Alternative mechanisms provided by Kubernetes such as the use of OIDC should be -implemented in place of client certificates. - - -**Audit:** - -```bash - -``` - - -## 3.2 Logging -### 3.2.1 Ensure that a minimal audit policy is created (Automated) - -**Result:** pass - -**Remediation:** -Create an audit policy file for your cluster. - - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--audit-policy-file' is present -``` - -**Returned Value**: - -```console -root 4643 4626 22 16:15 ? 00:00:46 kube-apiserver --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authorization-mode=Node,RBAC --audit-log-maxsize=100 --audit-log-format=json --requestheader-allowed-names=kube-apiserver-proxy-client --cloud-provider= --etcd-prefix=/registry --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --allow-privileged=true --service-account-lookup=true --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-policy-file=/etc/kubernetes/audit-policy.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --storage-backend=etcd3 --anonymous-auth=false --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --advertise-address=192.168.1.225 --audit-log-maxage=30 --etcd-servers=https://192.168.1.225:2379 --runtime-config=policy/v1beta1/podsecuritypolicy=true --bind-address=0.0.0.0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --insecure-port=0 --requestheader-group-headers=X-Remote-Group --secure-port=6443 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --requestheader-extra-headers-prefix=X-Remote-Extra- --profiling=false --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User - -``` -### 3.2.2 Ensure that the audit policy covers key security concerns (Manual) - -**Result:** warn - -**Remediation:** -Consider modification of the audit policy in use on the cluster to include these items, at a -minimum. - - -**Audit:** - -```bash - -``` - - -## 4.1 Worker Node Configuration Files -### 4.1.1 Ensure that the kubelet service file permissions are set to 644 or more restrictive (Automated) - -**Result:** notApplicable - -**Remediation:** -Cluster provisioned by RKE doesn’t require or maintain a configuration file for the kubelet service. -All configuration is passed in as arguments at container run time. - - -**Audit:** - -```bash -/bin/sh -c 'if test -e /etc/systemd/system/kubelet.service.d/10-kubeadm.conf; then stat -c permissions=%a /etc/systemd/system/kubelet.service.d/10-kubeadm.conf; fi' -``` - - -### 4.1.2 Ensure that the kubelet service file ownership is set to root:root (Automated) - -**Result:** notApplicable - -**Remediation:** -Cluster provisioned by RKE doesn’t require or maintain a configuration file for the kubelet service. -All configuration is passed in as arguments at container run time. - - -**Audit:** - -```bash -/bin/sh -c 'if test -e /etc/systemd/system/kubelet.service.d/10-kubeadm.conf; then stat -c %U:%G /etc/systemd/system/kubelet.service.d/10-kubeadm.conf; fi' -``` - - -### 4.1.3 If proxy kubeconfig file exists ensure permissions are set to 644 or more restrictive (Automated) - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, -chmod 644 $proykubeconfig - - -**Audit:** - -```bash -/bin/sh -c 'if test -e /node/etc/kubernetes/ssl/kubecfg-kube-proxy.yaml; then stat -c %a /node/etc/kubernetes/ssl/kubecfg-kube-proxy.yaml; fi' -``` - -**Expected Result**: - -```console -'644' is present OR '640' is present OR '600' is equal to '600' OR '444' is present OR '440' is present OR '400' is present OR '000' is present -``` - -**Returned Value**: - -```console -600 - -``` -### 4.1.4 Ensure that the proxy kubeconfig file ownership is set to root:root (Automated) - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, chown root:root /etc/kubernetes/ssl/kubecfg-kube-proxy.yaml - - -**Audit:** - -```bash -/bin/sh -c 'if test -e /etc/kubernetes/ssl/kubecfg-kube-proxy.yaml; then stat -c %U:%G /etc/kubernetes/ssl/kubecfg-kube-proxy.yaml; fi' -``` - -**Expected Result**: - -```console -'root:root' is not present OR '/etc/kubernetes/ssl/kubecfg-kube-proxy.yaml' is not present -``` - -### 4.1.5 Ensure that the --kubeconfig kubelet.conf file permissions are set to 644 or more restrictive (Automated) - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, -chmod 644 /etc/kubernetes/ssl/kubecfg-kube-node.yaml - - -**Audit:** - -```bash -/bin/sh -c 'if test -e /etc/kubernetes/ssl/kubecfg-kube-node.yaml; then stat -c permissions=%a /etc/kubernetes/ssl/kubecfg-kube-node.yaml; fi' -``` - -**Expected Result**: - -```console -'permissions' is not present -``` - -### 4.1.6 Ensure that the --kubeconfig kubelet.conf file ownership is set to root:root (Automated) - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, -chown root:root /etc/kubernetes/ssl/kubecfg-kube-node.yaml - - -**Audit:** - -```bash -/bin/sh -c 'if test -e /node/etc/kubernetes/ssl/kubecfg-kube-node.yaml; then stat -c %U:%G /node/etc/kubernetes/ssl/kubecfg-kube-node.yaml; fi' -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root - -``` -### 4.1.7 Ensure that the certificate authorities file permissions are set to 644 or more restrictive (Automated) - -**Result:** pass - -**Remediation:** -Run the following command to modify the file permissions of the ---client-ca-file chmod 644 - - -**Audit:** - -```bash -check_cafile_permissions.sh -``` - -**Expected Result**: - -```console -'permissions' is not present -``` - -**Audit Script:** -```bash -#!/usr/bin/env bash - -CAFILE=$(ps -ef | grep kubelet | grep -v apiserver | grep -- --client-ca-file= | awk -F '--client-ca-file=' '{print $2}' | awk '{print $1}') -if test -z $CAFILE; then CAFILE=$kubeletcafile; fi -if test -e $CAFILE; then stat -c permissions=%a $CAFILE; fi - -``` -### 4.1.8 Ensure that the client certificate authorities file ownership is set to root:root (Automated) - -**Result:** pass - -**Remediation:** -Run the following command to modify the ownership of the --client-ca-file. -chown root:root - - -**Audit:** - -```bash -check_cafile_ownership.sh -``` - -**Expected Result**: - -```console -'root:root' is not present -``` - -**Audit Script:** -```bash -#!/usr/bin/env bash - -CAFILE=$(ps -ef | grep kubelet | grep -v apiserver | grep -- --client-ca-file= | awk -F '--client-ca-file=' '{print $2}' | awk '{print $1}') -if test -z $CAFILE; then CAFILE=$kubeletcafile; fi -if test -e $CAFILE; then stat -c %U:%G $CAFILE; fi - -``` -### 4.1.9 Ensure that the kubelet --config configuration file has permissions set to 644 or more restrictive (Automated) - -**Result:** notApplicable - -**Remediation:** -Run the following command (using the config file location identified in the Audit step) -chmod 644 /var/lib/kubelet/config.yaml - -Clusters provisioned by RKE doesn’t require or maintain a configuration file for the kubelet. -All configuration is passed in as arguments at container run time. - - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then stat -c permissions=%a /var/lib/kubelet/config.yaml; fi' -``` - - -### 4.1.10 Ensure that the kubelet --config configuration file ownership is set to root:root (Automated) - -**Result:** notApplicable - -**Remediation:** -Run the following command (using the config file location identified in the Audit step) -chown root:root /var/lib/kubelet/config.yaml - -Clusters provisioned by RKE doesn’t require or maintain a configuration file for the kubelet. -All configuration is passed in as arguments at container run time. - - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then stat -c %U:%G /var/lib/kubelet/config.yaml; fi' -``` - - -## 4.2 Kubelet -### 4.2.1 Ensure that the anonymous-auth argument is set to false (Automated) - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set authentication: anonymous: enabled to -false. -If using executable arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. ---anonymous-auth=false -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Expected Result**: - -```console -'' is not present -``` - -### 4.2.2 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated) - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set authorization: mode to Webhook. If -using executable arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_AUTHZ_ARGS variable. ---authorization-mode=Webhook -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Expected Result**: - -```console -'' is not present -``` - -### 4.2.3 Ensure that the --client-ca-file argument is set as appropriate (Automated) - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set authentication: x509: clientCAFile to -the location of the client CA file. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_AUTHZ_ARGS variable. ---client-ca-file= -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Expected Result**: - -```console -'' is not present -``` - -### 4.2.4 Ensure that the --read-only-port argument is set to 0 (Automated) - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set readOnlyPort to 0. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. ---read-only-port=0 -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Expected Result**: - -```console -'' is not present OR '' is not present -``` - -### 4.2.5 Ensure that the --streaming-connection-idle-timeout argument is not set to 0 (Automated) - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set streamingConnectionIdleTimeout to a -value other than 0. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. ---streaming-connection-idle-timeout=5m -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Expected Result**: - -```console -'30m' is not equal to '0' OR '--streaming-connection-idle-timeout' is not present -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD -root 5103 5086 7 16:16 ? 00:00:12 kubelet --resolv-conf=/etc/resolv.conf --read-only-port=0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --feature-gates=RotateKubeletServerCertificate=true --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --make-iptables-util-chains=true --streaming-connection-idle-timeout=30m --cluster-dns=10.43.0.10 --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-192-168-1-225-key.pem --address=0.0.0.0 --cni-bin-dir=/opt/cni/bin --anonymous-auth=false --protect-kernel-defaults=true --cloud-provider= --hostname-override=cis-aio-0 --fail-swap-on=false --cgroups-per-qos=True --authentication-token-webhook=true --event-qps=0 --v=2 --pod-infra-container-image=rancher/pause:3.1 --authorization-mode=Webhook --network-plugin=cni --cluster-domain=cluster.local --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --volume-plugin-dir=/var/lib/kubelet/volumeplugins --cni-conf-dir=/etc/cni/net.d --root-dir=/var/lib/kubelet --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-192-168-1-225.pem --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf - -``` -### 4.2.6 Ensure that the --protect-kernel-defaults argument is set to true (Automated) - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set protectKernelDefaults: true. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. ---protect-kernel-defaults=true -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Expected Result**: - -```console -'' is not present -``` - -### 4.2.7 Ensure that the --make-iptables-util-chains argument is set to true (Automated) - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set makeIPTablesUtilChains: true. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -remove the --make-iptables-util-chains argument from the -KUBELET_SYSTEM_PODS_ARGS variable. -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Expected Result**: - -```console -'' is not present OR '' is not present -``` - -### 4.2.8 Ensure that the --hostname-override argument is not set (Manual) - -**Result:** notApplicable - -**Remediation:** -Edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf -on each worker node and remove the --hostname-override argument from the -KUBELET_SYSTEM_PODS_ARGS variable. -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - -Clusters provisioned by RKE set the --hostname-override to avoid any hostname configuration errors - - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - - -### 4.2.9 Ensure that the --event-qps argument is set to 0 or a level which ensures appropriate event capture (Automated) - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set eventRecordQPS: to an appropriate level. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Expected Result**: - -```console -'' is not present -``` - -### 4.2.10 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Automated) - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set tlsCertFile to the location -of the certificate file to use to identify this Kubelet, and tlsPrivateKeyFile -to the location of the corresponding private key file. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameters in KUBELET_CERTIFICATE_ARGS variable. ---tls-cert-file= ---tls-private-key-file= -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Expected Result**: - -```console -'' is not present AND '' is not present -``` - -### 4.2.11 Ensure that the --rotate-certificates argument is not set to false (Automated) - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to add the line rotateCertificates: true or -remove it altogether to use the default value. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -remove --rotate-certificates=false argument from the KUBELET_CERTIFICATE_ARGS -variable. -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Expected Result**: - -```console -'--rotate-certificates' is not present OR '--rotate-certificates' is not present -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD -root 5103 5086 6 16:16 ? 00:00:12 kubelet --resolv-conf=/etc/resolv.conf --read-only-port=0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --feature-gates=RotateKubeletServerCertificate=true --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --make-iptables-util-chains=true --streaming-connection-idle-timeout=30m --cluster-dns=10.43.0.10 --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-192-168-1-225-key.pem --address=0.0.0.0 --cni-bin-dir=/opt/cni/bin --anonymous-auth=false --protect-kernel-defaults=true --cloud-provider= --hostname-override=cis-aio-0 --fail-swap-on=false --cgroups-per-qos=True --authentication-token-webhook=true --event-qps=0 --v=2 --pod-infra-container-image=rancher/pause:3.1 --authorization-mode=Webhook --network-plugin=cni --cluster-domain=cluster.local --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --volume-plugin-dir=/var/lib/kubelet/volumeplugins --cni-conf-dir=/etc/cni/net.d --root-dir=/var/lib/kubelet --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-192-168-1-225.pem --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf - -``` -### 4.2.12 Verify that the RotateKubeletServerCertificate argument is set to true (Automated) - -**Result:** notApplicable - -**Remediation:** -Edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf -on each worker node and set the below parameter in KUBELET_CERTIFICATE_ARGS variable. ---feature-gates=RotateKubeletServerCertificate=true -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - -Clusters provisioned by RKE handles certificate rotation directly through RKE. - - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - - -### 4.2.13 Ensure that the Kubelet only makes use of Strong Cryptographic Ciphers (Automated) - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set TLSCipherSuites: to -TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 -or to a subset of these values. -If using executable arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the --tls-cipher-suites parameter as follows, or to a subset of these values. ---tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Expected Result**: - -```console -'' is not present -``` - -## 5.1 RBAC and Service Accounts -### 5.1.1 Ensure that the cluster-admin role is only used where required (Manual) - -**Result:** warn - -**Remediation:** -Identify all clusterrolebindings to the cluster-admin role. Check if they are used and -if they need this role or if they could use a role with fewer privileges. -Where possible, first bind users to a lower privileged role and then remove the -clusterrolebinding to the cluster-admin role : -kubectl delete clusterrolebinding [name] - - -**Audit:** - -```bash - -``` - - -### 5.1.2 Minimize access to secrets (Manual) - -**Result:** warn - -**Remediation:** -Where possible, remove get, list and watch access to secret objects in the cluster. - - -**Audit:** - -```bash - -``` - - -### 5.1.3 Minimize wildcard use in Roles and ClusterRoles (Manual) - -**Result:** warn - -**Remediation:** -Where possible replace any use of wildcards in clusterroles and roles with specific -objects or actions. - - -**Audit:** - -```bash - -``` - - -### 5.1.4 Minimize access to create pods (Manual) - -**Result:** warn - -**Remediation:** -Where possible, remove create access to pod objects in the cluster. - - -**Audit:** - -```bash - -``` - - -### 5.1.5 Ensure that default service accounts are not actively used. (Automated) - -**Result:** pass - -**Remediation:** -Create explicit service accounts wherever a Kubernetes workload requires specific access -to the Kubernetes API server. -Modify the configuration of each default service account to include this value -automountServiceAccountToken: false - - -**Audit:** - -```bash -check_for_default_sa.sh -``` - -**Expected Result**: - -```console -'true' is equal to 'true' -``` - -**Audit Script:** -```bash -#!/bin/bash - -set -eE - -handle_error() { - echo "false" -} - -trap 'handle_error' ERR - -count_sa=$(kubectl get serviceaccounts --all-namespaces -o json | jq -r '.items[] | select(.metadata.name=="default") | select((.automountServiceAccountToken == null) or (.automountServiceAccountToken == true))' | jq .metadata.namespace | wc -l) -if [[ ${count_sa} -gt 0 ]]; then - echo "false" - exit -fi - -for ns in $(kubectl get ns --no-headers -o custom-columns=":metadata.name") -do - for result in $(kubectl get clusterrolebinding,rolebinding -n $ns -o json | jq -r '.items[] | select((.subjects[].kind=="ServiceAccount" and .subjects[].name=="default") or (.subjects[].kind=="Group" and .subjects[].name=="system:serviceaccounts"))' | jq -r '"\(.roleRef.kind),\(.roleRef.name)"') - do - read kind name <<<$(IFS=","; echo $result) - resource_count=$(kubectl get $kind $name -n $ns -o json | jq -r '.rules[] | select(.resources[] != "podsecuritypolicies")' | wc -l) - if [[ ${resource_count} -gt 0 ]]; then - echo "false" - exit - fi - done -done - - -echo "true" -``` -**Returned Value**: - -```console -true - -``` -### 5.1.6 Ensure that Service Account Tokens are only mounted where necessary (Manual) - -**Result:** warn - -**Remediation:** -Modify the definition of pods and service accounts which do not need to mount service -account tokens to disable it. - - -**Audit:** - -```bash - -``` - - -## 5.2 Pod Security Policies -### 5.2.1 Minimize the admission of privileged containers (Manual) - -**Result:** warn - -**Remediation:** -Create a PSP as described in the Kubernetes documentation, ensuring that -the .spec.privileged field is omitted or set to false. - - -**Audit:** - -```bash - -``` - - -### 5.2.2 Minimize the admission of containers wishing to share the host process ID namespace (Automated) - -**Result:** pass - -**Remediation:** -Create a PSP as described in the Kubernetes documentation, ensuring that the -.spec.hostPID field is omitted or set to false. - - -**Audit:** - -```bash -kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.hostPID == null) or (.spec.hostPID == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' -``` - -**Expected Result**: - -```console -1 is greater than 0 -``` - -**Returned Value**: - -```console ---count=1 - -``` -### 5.2.3 Minimize the admission of containers wishing to share the host IPC namespace (Automated) - -**Result:** pass - -**Remediation:** -Create a PSP as described in the Kubernetes documentation, ensuring that the -.spec.hostIPC field is omitted or set to false. - - -**Audit:** - -```bash -kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.hostIPC == null) or (.spec.hostIPC == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' -``` - -**Expected Result**: - -```console -1 is greater than 0 -``` - -**Returned Value**: - -```console ---count=1 - -``` -### 5.2.4 Minimize the admission of containers wishing to share the host network namespace (Automated) - -**Result:** pass - -**Remediation:** -Create a PSP as described in the Kubernetes documentation, ensuring that the -.spec.hostNetwork field is omitted or set to false. - - -**Audit:** - -```bash -kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.hostNetwork == null) or (.spec.hostNetwork == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' -``` - -**Expected Result**: - -```console -1 is greater than 0 -``` - -**Returned Value**: - -```console ---count=1 - -``` -### 5.2.5 Minimize the admission of containers with allowPrivilegeEscalation (Automated) - -**Result:** pass - -**Remediation:** -Create a PSP as described in the Kubernetes documentation, ensuring that the -.spec.allowPrivilegeEscalation field is omitted or set to false. - - -**Audit:** - -```bash -kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.allowPrivilegeEscalation == null) or (.spec.allowPrivilegeEscalation == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' -``` - -**Expected Result**: - -```console -1 is greater than 0 -``` - -**Returned Value**: - -```console ---count=1 - -``` -### 5.2.6 Minimize the admission of root containers (Manual) - -**Result:** warn - -**Remediation:** -Create a PSP as described in the Kubernetes documentation, ensuring that the -.spec.runAsUser.rule is set to either MustRunAsNonRoot or MustRunAs with the range of -UIDs not including 0. - - -**Audit:** - -```bash - -``` - - -### 5.2.7 Minimize the admission of containers with the NET_RAW capability (Manual) - -**Result:** warn - -**Remediation:** -Create a PSP as described in the Kubernetes documentation, ensuring that the -.spec.requiredDropCapabilities is set to include either NET_RAW or ALL. - - -**Audit:** - -```bash - -``` - - -### 5.2.8 Minimize the admission of containers with added capabilities (Manual) - -**Result:** warn - -**Remediation:** -Ensure that allowedCapabilities is not present in PSPs for the cluster unless -it is set to an empty array. - - -**Audit:** - -```bash - -``` - - -### 5.2.9 Minimize the admission of containers with capabilities assigned (Manual) - -**Result:** warn - -**Remediation:** -Review the use of capabilites in applications runnning on your cluster. Where a namespace -contains applicaions which do not require any Linux capabities to operate consider adding -a PSP which forbids the admission of containers which do not drop all capabilities. - - -**Audit:** - -```bash - -``` - - -## 5.3 Network Policies and CNI -### 5.3.1 Ensure that the CNI in use supports Network Policies (Manual) - -**Result:** warn - -**Remediation:** -If the CNI plugin in use does not support network policies, consideration should be given to -making use of a different plugin, or finding an alternate mechanism for restricting traffic -in the Kubernetes cluster. - - -**Audit:** - -```bash - -``` - - -### 5.3.2 Ensure that all Namespaces have Network Policies defined (Automated) - -**Result:** pass - -**Remediation:** -Follow the documentation and create NetworkPolicy objects as you need them. - - -**Audit:** - -```bash -check_for_network_policies.sh -``` - -**Expected Result**: - -```console -'true' is equal to 'true' -``` - -**Audit Script:** -```bash -#!/bin/bash - -set -eE - -handle_error() { - echo "false" -} - -trap 'handle_error' ERR - -for namespace in $(kubectl get namespaces --all-namespaces -o json | jq -r '.items[].metadata.name'); do - policy_count=$(kubectl get networkpolicy -n ${namespace} -o json | jq '.items | length') - if [[ ${policy_count} -eq 0 ]]; then - echo "false" - exit - fi -done - -echo "true" - -``` -**Returned Value**: - -```console -true - -``` -## 5.4 Secrets Management -### 5.4.1 Prefer using secrets as files over secrets as environment variables (Manual) - -**Result:** warn - -**Remediation:** -if possible, rewrite application code to read secrets from mounted secret files, rather than -from environment variables. - - -**Audit:** - -```bash - -``` - - -### 5.4.2 Consider external secret storage (Manual) - -**Result:** warn - -**Remediation:** -Refer to the secrets management options offered by your cloud provider or a third-party -secrets management solution. - - -**Audit:** - -```bash - -``` - - -## 5.5 Extensible Admission Control -### 5.5.1 Configure Image Provenance using ImagePolicyWebhook admission controller (Manual) - -**Result:** warn - -**Remediation:** -Follow the Kubernetes documentation and setup image provenance. - - -**Audit:** - -```bash - -``` - - -## 5.7 General Policies -### 5.7.1 Create administrative boundaries between resources using namespaces (Manual) - -**Result:** warn - -**Remediation:** -Follow the documentation and create namespaces for objects in your deployment as you need -them. - - -**Audit:** - -```bash - -``` - - -### 5.7.2 Ensure that the seccomp profile is set to docker/default in your pod definitions (Manual) - -**Result:** warn - -**Remediation:** -Seccomp is an alpha feature currently. By default, all alpha features are disabled. So, you -would need to enable alpha features in the apiserver by passing "--feature- -gates=AllAlpha=true" argument. -Edit the /etc/kubernetes/apiserver file on the master node and set the KUBE_API_ARGS -parameter to "--feature-gates=AllAlpha=true" -KUBE_API_ARGS="--feature-gates=AllAlpha=true" -Based on your system, restart the kube-apiserver service. For example: -systemctl restart kube-apiserver.service -Use annotations to enable the docker/default seccomp profile in your pod definitions. An -example is as below: -apiVersion: v1 -kind: Pod -metadata: - name: trustworthy-pod - annotations: - seccomp.security.alpha.kubernetes.io/pod: docker/default -spec: - containers: - - name: trustworthy-container - image: sotrustworthy:latest - - -**Audit:** - -```bash - -``` - - -### 5.7.3 Apply Security Context to Your Pods and Containers (Manual) - -**Result:** warn - -**Remediation:** -Follow the Kubernetes documentation and apply security contexts to your pods. For a -suggested list of security contexts, you may refer to the CIS Security Benchmark for Docker -Containers. - - -**Audit:** - -```bash - -``` - - -### 5.7.4 The default namespace should not be used (Automated) - -**Result:** pass - -**Remediation:** -Ensure that namespaces are created to allow for appropriate segregation of Kubernetes -resources and that all new resources are created in a specific namespace. - - -**Audit:** - -```bash -check_for_default_ns.sh -``` - -**Expected Result**: - -```console -'true' is equal to 'true' -``` - -**Audit Script:** -```bash -#!/bin/bash - -set -eE - -handle_error() { - echo "false" -} - -trap 'handle_error' ERR - -count=$(kubectl get all -n default -o json | jq .items[] | jq -r 'select((.metadata.name!="kubernetes"))' | jq .metadata.name | wc -l) -if [[ ${count} -gt 0 ]]; then - echo "false" - exit -fi - -echo "true" - - -``` -**Returned Value**: - -```console -true - -``` diff --git a/content/rancher/v2.6/en/security/rancher-2.5/1.6-hardening-2.5/_index.md b/content/rancher/v2.6/en/security/rancher-2.5/1.6-hardening-2.5/_index.md deleted file mode 100644 index 78f2763e57c..00000000000 --- a/content/rancher/v2.6/en/security/rancher-2.5/1.6-hardening-2.5/_index.md +++ /dev/null @@ -1,570 +0,0 @@ ---- -title: Hardening Guide with CIS 1.6 Benchmark -weight: 100 ---- - -This document provides prescriptive guidance for hardening a production installation of a RKE cluster to be used with Rancher v2.5.4. It outlines the configurations and controls required to address Kubernetes benchmark controls from the Center for Information Security (CIS). - -> This hardening guide describes how to secure the nodes in your cluster, and it is recommended to follow this guide before installing Kubernetes. - -This hardening guide is intended to be used for RKE clusters and associated with specific versions of the CIS Kubernetes Benchmark, Kubernetes, and Rancher: - - Rancher Version | CIS Benchmark Version | Kubernetes Version -----------------|-----------------------|------------------ - Rancher v2.5.4 | Benchmark 1.6 | Kubernetes v1.18 - -[Click here to download a PDF version of this document](https://releases.rancher.com/documents/security/2.5/Rancher_Hardening_Guide_CIS_1.6.pdf) - -### Overview - -This document provides prescriptive guidance for hardening a RKE cluster to be used for installing Rancher v2.5.4 with Kubernetes v1.18 or provisioning a RKE cluster with Kubernetes v1.18 to be used within Rancher v2.5.4. It outlines the configurations required to address Kubernetes benchmark controls from the Center for Information Security (CIS). - -For more detail about evaluating a hardened cluster against the official CIS benchmark, refer to the [CIS 1.6 Benchmark - Self-Assessment Guide - Rancher v2.5.4]({{< baseurl >}}/rancher/v2.6/en/security/rancher-2.5/1.6-benchmark-2.5/). - -#### Known Issues - -- Rancher **exec shell** and **view logs** for pods are **not** functional in a CIS 1.6 hardened setup when only public IP is provided when registering custom nodes. This functionality requires a private IP to be provided when registering the custom nodes. -- When setting the `default_pod_security_policy_template_id:` to `restricted` Rancher creates **RoleBindings** and **ClusterRoleBindings** on the default service accounts. The CIS 1.6 5.1.5 check requires the default service accounts have no roles or cluster roles bound to it apart from the defaults. In addition the default service accounts should be configured such that it does not provide a service account token and does not have any explicit rights assignments. - -Migration Rancher from 2.4 to 2.5. Addons were removed in HG 2.5, and therefore namespaces on migration may be not created on the downstream clusters. Pod may fail to run because of missing namesapce like ingress-nginx, cattle-system. - -### Configure Kernel Runtime Parameters - -The following `sysctl` configuration is recommended for all nodes type in the cluster. Set the following parameters in `/etc/sysctl.d/90-kubelet.conf`: - -```ini -vm.overcommit_memory=1 -vm.panic_on_oom=0 -kernel.panic=10 -kernel.panic_on_oops=1 -kernel.keys.root_maxbytes=25000000 -``` - -Run `sysctl -p /etc/sysctl.d/90-kubelet.conf` to enable the settings. - -### Configure `etcd` user and group -A user account and group for the **etcd** service is required to be setup before installing RKE. The **uid** and **gid** for the **etcd** user will be used in the RKE **config.yml** to set the proper permissions for files and directories during installation time. - -#### create `etcd` user and group -To create the **etcd** group run the following console commands. - -The commands below use `52034` for **uid** and **gid** are for example purposes. Any valid unused **uid** or **gid** could also be used in lieu of `52034`. - -```bash -groupadd --gid 52034 etcd -useradd --comment "etcd service account" --uid 52034 --gid 52034 etcd -``` - -Update the RKE **config.yml** with the **uid** and **gid** of the **etcd** user: - -```yaml -services: - etcd: - gid: 52034 - uid: 52034 -``` - -#### Set `automountServiceAccountToken` to `false` for `default` service accounts -Kubernetes provides a default service account which is used by cluster workloads where no specific service account is assigned to the pod. Where access to the Kubernetes API from a pod is required, a specific service account should be created for that pod, and rights granted to that service account. The default service account should be configured such that it does not provide a service account token and does not have any explicit rights assignments. - -For each namespace including **default** and **kube-system** on a standard RKE install the **default** service account must include this value: - -```yaml -automountServiceAccountToken: false -``` - -Save the following yaml to a file called `account_update.yaml` - -```yaml -apiVersion: v1 -kind: ServiceAccount -metadata: - name: default -automountServiceAccountToken: false -``` - -Create a bash script file called `account_update.sh`. Be sure to `chmod +x account_update.sh` so the script has execute permissions. - -```bash -#!/bin/bash -e - -for namespace in $(kubectl get namespaces -A -o json | jq -r '.items[].metadata.name'); do - kubectl patch serviceaccount default -n ${namespace} -p "$(cat account_update.yaml)" -done -``` - -### Ensure that all Namespaces have Network Policies defined - -Running different applications on the same Kubernetes cluster creates a risk of one -compromised application attacking a neighboring application. Network segmentation is -important to ensure that containers can communicate only with those they are supposed -to. A network policy is a specification of how selections of pods are allowed to -communicate with each other and other network endpoints. - -Network Policies are namespace scoped. When a network policy is introduced to a given -namespace, all traffic not allowed by the policy is denied. However, if there are no network -policies in a namespace all traffic will be allowed into and out of the pods in that -namespace. To enforce network policies, a CNI (container network interface) plugin must be enabled. -This guide uses [canal](https://github.com/projectcalico/canal) to provide the policy enforcement. -Additional information about CNI providers can be found -[here](https://rancher.com/blog/2019/2019-03-21-comparing-kubernetes-cni-providers-flannel-calico-canal-and-weave/) - -Once a CNI provider is enabled on a cluster a default network policy can be applied. For reference purposes a -**permissive** example is provide below. If you want to allow all traffic to all pods in a namespace -(even if policies are added that cause some pods to be treated as “isolated”), -you can create a policy that explicitly allows all traffic in that namespace. Save the following `yaml` as -`default-allow-all.yaml`. Additional [documentation](https://kubernetes.io/docs/concepts/services-networking/network-policies/) -about network policies can be found on the Kubernetes site. - -> This `NetworkPolicy` is not recommended for production use - -```yaml ---- -apiVersion: networking.k8s.io/v1 -kind: NetworkPolicy -metadata: - name: default-allow-all -spec: - podSelector: {} - ingress: - - {} - egress: - - {} - policyTypes: - - Ingress - - Egress -``` - -Create a bash script file called `apply_networkPolicy_to_all_ns.sh`. Be sure to -`chmod +x apply_networkPolicy_to_all_ns.sh` so the script has execute permissions. - -```bash -#!/bin/bash -e - -for namespace in $(kubectl get namespaces -A -o json | jq -r '.items[].metadata.name'); do - kubectl apply -f default-allow-all.yaml -n ${namespace} -done -``` - -Execute this script to apply the `default-allow-all.yaml` the **permissive** `NetworkPolicy` to all namespaces. - -### Reference Hardened RKE `cluster.yml` configuration - -The reference `cluster.yml` is used by the RKE CLI that provides the configuration needed to achieve a hardened install -of Rancher Kubernetes Engine (RKE). Install [documentation](https://rancher.com/docs/rke/latest/en/installation/) is -provided with additional details about the configuration items. This reference `cluster.yml` does not include the required **nodes** directive which will vary depending on your environment. Documentation for node configuration can be found here: https://rancher.com/docs/rke/latest/en/config-options/nodes - - -```yaml -# If you intend to deploy Kubernetes in an air-gapped environment, -# please consult the documentation on how to configure custom RKE images. -# https://rancher.com/docs/rke/latest/en/installation/ - -# the nodes directive is required and will vary depending on your environment -# documentation for node configuration can be found here: -# https://rancher.com/docs/rke/latest/en/config-options/nodes -nodes: [] -services: - etcd: - image: "" - extra_args: {} - extra_binds: [] - extra_env: [] - win_extra_args: {} - win_extra_binds: [] - win_extra_env: [] - external_urls: [] - ca_cert: "" - cert: "" - key: "" - path: "" - uid: 52034 - gid: 52034 - snapshot: false - retention: "" - creation: "" - backup_config: null - kube-api: - image: "" - extra_args: {} - extra_binds: [] - extra_env: [] - win_extra_args: {} - win_extra_binds: [] - win_extra_env: [] - service_cluster_ip_range: "" - service_node_port_range: "" - pod_security_policy: true - always_pull_images: false - secrets_encryption_config: - enabled: true - custom_config: null - audit_log: - enabled: true - configuration: null - admission_configuration: null - event_rate_limit: - enabled: true - configuration: null - kube-controller: - image: "" - extra_args: - feature-gates: RotateKubeletServerCertificate=true - extra_binds: [] - extra_env: [] - win_extra_args: {} - win_extra_binds: [] - win_extra_env: [] - cluster_cidr: "" - service_cluster_ip_range: "" - scheduler: - image: "" - extra_args: {} - extra_binds: [] - extra_env: [] - win_extra_args: {} - win_extra_binds: [] - win_extra_env: [] - kubelet: - image: "" - extra_args: - feature-gates: RotateKubeletServerCertificate=true - protect-kernel-defaults: "true" - tls-cipher-suites: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 - extra_binds: [] - extra_env: [] - win_extra_args: {} - win_extra_binds: [] - win_extra_env: [] - cluster_domain: cluster.local - infra_container_image: "" - cluster_dns_server: "" - fail_swap_on: false - generate_serving_certificate: true - kubeproxy: - image: "" - extra_args: {} - extra_binds: [] - extra_env: [] - win_extra_args: {} - win_extra_binds: [] - win_extra_env: [] -network: - plugin: "" - options: {} - mtu: 0 - node_selector: {} - update_strategy: null -authentication: - strategy: "" - sans: [] - webhook: null -addons: | - apiVersion: policy/v1beta1 - kind: PodSecurityPolicy - metadata: - name: restricted - spec: - requiredDropCapabilities: - - NET_RAW - privileged: false - allowPrivilegeEscalation: false - defaultAllowPrivilegeEscalation: false - fsGroup: - rule: RunAsAny - runAsUser: - rule: MustRunAsNonRoot - seLinux: - rule: RunAsAny - supplementalGroups: - rule: RunAsAny - volumes: - - emptyDir - - secret - - persistentVolumeClaim - - downwardAPI - - configMap - - projected - --- - apiVersion: rbac.authorization.k8s.io/v1 - kind: ClusterRole - metadata: - name: psp:restricted - rules: - - apiGroups: - - extensions - resourceNames: - - restricted - resources: - - podsecuritypolicies - verbs: - - use - --- - apiVersion: rbac.authorization.k8s.io/v1 - kind: ClusterRoleBinding - metadata: - name: psp:restricted - roleRef: - apiGroup: rbac.authorization.k8s.io - kind: ClusterRole - name: psp:restricted - subjects: - - apiGroup: rbac.authorization.k8s.io - kind: Group - name: system:serviceaccounts - - apiGroup: rbac.authorization.k8s.io - kind: Group - name: system:authenticated - --- - apiVersion: networking.k8s.io/v1 - kind: NetworkPolicy - metadata: - name: default-allow-all - spec: - podSelector: {} - ingress: - - {} - egress: - - {} - policyTypes: - - Ingress - - Egress - --- - apiVersion: v1 - kind: ServiceAccount - metadata: - name: default - automountServiceAccountToken: false -addons_include: [] -system_images: - etcd: "" - alpine: "" - nginx_proxy: "" - cert_downloader: "" - kubernetes_services_sidecar: "" - kubedns: "" - dnsmasq: "" - kubedns_sidecar: "" - kubedns_autoscaler: "" - coredns: "" - coredns_autoscaler: "" - nodelocal: "" - kubernetes: "" - flannel: "" - flannel_cni: "" - calico_node: "" - calico_cni: "" - calico_controllers: "" - calico_ctl: "" - calico_flexvol: "" - canal_node: "" - canal_cni: "" - canal_controllers: "" - canal_flannel: "" - canal_flexvol: "" - weave_node: "" - weave_cni: "" - pod_infra_container: "" - ingress: "" - ingress_backend: "" - metrics_server: "" - windows_pod_infra_container: "" -ssh_key_path: "" -ssh_cert_path: "" -ssh_agent_auth: false -authorization: - mode: "" - options: {} -ignore_docker_version: false -kubernetes_version: v1.18.12-rancher1-1 -private_registries: [] -ingress: - provider: "" - options: {} - node_selector: {} - extra_args: {} - dns_policy: "" - extra_envs: [] - extra_volumes: [] - extra_volume_mounts: [] - update_strategy: null - http_port: 0 - https_port: 0 - network_mode: "" -cluster_name: -cloud_provider: - name: "" -prefix_path: "" -win_prefix_path: "" -addon_job_timeout: 0 -bastion_host: - address: "" - port: "" - user: "" - ssh_key: "" - ssh_key_path: "" - ssh_cert: "" - ssh_cert_path: "" -monitoring: - provider: "" - options: {} - node_selector: {} - update_strategy: null - replicas: null -restore: - restore: false - snapshot_name: "" -dns: null -upgrade_strategy: - max_unavailable_worker: "" - max_unavailable_controlplane: "" - drain: null - node_drain_input: null -``` - -### Reference Hardened RKE Template configuration - -The reference RKE Template provides the configuration needed to achieve a hardened install of Kubenetes. -RKE Templates are used to provision Kubernetes and define Rancher settings. Follow the Rancher -[documentaion](https://rancher.com/docs/rancher/v2.6/en/installation) for additional installation and RKE Template details. - -```yaml -# -# Cluster Config -# -default_pod_security_policy_template_id: restricted -docker_root_dir: /var/lib/docker -enable_cluster_alerting: false -enable_cluster_monitoring: false -enable_network_policy: true -# -# Rancher Config -# -rancher_kubernetes_engine_config: - addon_job_timeout: 45 - ignore_docker_version: true - kubernetes_version: v1.18.12-rancher1-1 -# -# If you are using calico on AWS -# -# network: -# plugin: calico -# calico_network_provider: -# cloud_provider: aws -# -# # To specify flannel interface -# -# network: -# plugin: flannel -# flannel_network_provider: -# iface: eth1 -# -# # To specify flannel interface for canal plugin -# -# network: -# plugin: canal -# canal_network_provider: -# iface: eth1 -# - network: - mtu: 0 - plugin: canal - rotate_encryption_key: false -# -# services: -# kube-api: -# service_cluster_ip_range: 10.43.0.0/16 -# kube-controller: -# cluster_cidr: 10.42.0.0/16 -# service_cluster_ip_range: 10.43.0.0/16 -# kubelet: -# cluster_domain: cluster.local -# cluster_dns_server: 10.43.0.10 -# - services: - etcd: - backup_config: - enabled: false - interval_hours: 12 - retention: 6 - safe_timestamp: false - creation: 12h - extra_args: - election-timeout: '5000' - heartbeat-interval: '500' - gid: 52034 - retention: 72h - snapshot: false - uid: 52034 - kube_api: - always_pull_images: false - audit_log: - enabled: true - event_rate_limit: - enabled: true - pod_security_policy: true - secrets_encryption_config: - enabled: true - service_node_port_range: 30000-32767 - kube_controller: - extra_args: - feature-gates: RotateKubeletServerCertificate=true - kubelet: - extra_args: - feature-gates: RotateKubeletServerCertificate=true - protect-kernel-defaults: 'true' - tls-cipher-suites: >- - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 - fail_swap_on: false - generate_serving_certificate: true - ssh_agent_auth: false - upgrade_strategy: - max_unavailable_controlplane: '1' - max_unavailable_worker: 10% -windows_prefered_cluster: false -``` - -### Hardened Reference Ubuntu 20.04 LTS **cloud-config**: - -The reference **cloud-config** is generally used in cloud infrastructure environments to allow for -configuration management of compute instances. The reference config configures Ubuntu operating system level settings -needed before installing kubernetes. - -```yaml -#cloud-config -apt: - sources: - docker.list: - source: deb [arch=amd64] http://download.docker.com/linux/ubuntu $RELEASE stable - keyid: 9DC858229FC7DD38854AE2D88D81803C0EBFCD88 -system_info: - default_user: - groups: - - docker -write_files: -- path: "/etc/apt/preferences.d/docker" - owner: root:root - permissions: '0600' - content: | - Package: docker-ce - Pin: version 5:19* - Pin-Priority: 800 -- path: "/etc/sysctl.d/90-kubelet.conf" - owner: root:root - permissions: '0644' - content: | - vm.overcommit_memory=1 - vm.panic_on_oom=0 - kernel.panic=10 - kernel.panic_on_oops=1 - kernel.keys.root_maxbytes=25000000 -package_update: true -packages: -- docker-ce -- docker-ce-cli -- containerd.io -runcmd: -- sysctl -p /etc/sysctl.d/90-kubelet.conf -- groupadd --gid 52034 etcd -- useradd --comment "etcd service account" --uid 52034 --gid 52034 etcd -``` diff --git a/content/rancher/v2.6/en/security/rancher-2.5/_index.md b/content/rancher/v2.6/en/security/rancher-2.5/_index.md deleted file mode 100644 index 7282a7d813a..00000000000 --- a/content/rancher/v2.6/en/security/rancher-2.5/_index.md +++ /dev/null @@ -1,55 +0,0 @@ ---- -title: Self-Assessment and Hardening Guides for Rancher v2.5 -shortTitle: Rancher v2.5 Guides -weight: 1 ---- - -Rancher v2.5 introduced the capability to deploy Rancher on any Kubernetes cluster. For that reason, we now provide separate security hardening guides for Rancher deployments on each of Rancher's Kubernetes distributions. - -- [Rancher Kubernetes Distributions](#rancher-kubernetes-distributions) -- [Hardening Guides and Benchmark Versions](#hardening-guides-and-benchmark-versions) - - [RKE Guides](#rke-guides) - - [RKE2 Guides](#rke2-guides) - - [K3s Guides](#k3s) -- [Rancher with SELinux](#rancher-with-selinux) - -# Rancher Kubernetes Distributions - -Rancher has the following Kubernetes distributions: - -- [**RKE,**]({{}}/rke/latest/en/) Rancher Kubernetes Engine, is a CNCF-certified Kubernetes distribution that runs entirely within Docker containers. -- [**K3s,**]({{}}/k3s/latest/en/) is a fully conformant, lightweight Kubernetes distribution. It is easy to install, with half the memory of upstream Kubernetes, all in a binary of less than 100 MB. -- [**RKE2**](https://docs.rke2.io/) is a fully conformant Kubernetes distribution that focuses on security and compliance within the U.S. Federal Government sector. - -To harden a Kubernetes cluster outside of Rancher's distributions, refer to your Kubernetes provider docs. - -# Hardening Guides and Benchmark Versions - -These guides have been tested along with the Rancher v2.5 release. Each self-assessment guide is accompanied with a hardening guide and tested on a specific Kubernetes version and CIS benchmark version. If a CIS benchmark has not been validated for your Kubernetes version, you can choose to use the existing guides until a newer version is added. - -### RKE Guides - -Kubernetes Version | CIS Benchmark Version | Self Assessment Guide | Hardening Guides ----|---|---|--- -Kubernetes v1.15+ | CIS v1.5 | [Link](./1.5-benchmark-2.5) | [Link](./1.5-hardening-2.5) -Kubernetes v1.18+ | CIS v1.6 | [Link](./1.6-benchmark-2.5) | [Link](./1.6-hardening-2.5) - -### RKE2 Guides - -Kubernetes Version | CIS Benchmark Version | Self Assessment Guide | Hardening Guides ----|---|---|--- -Kubernetes v1.18 | CIS v1.5 | [Link](https://docs.rke2.io/security/cis_self_assessment15/) | [Link](https://docs.rke2.io/security/hardening_guide/) -Kubernetes v1.20 | CIS v1.6 | [Link](https://docs.rke2.io/security/cis_self_assessment16/) | [Link](https://docs.rke2.io/security/hardening_guide/) - -### K3s Guides - -Kubernetes Version | CIS Benchmark Version | Self Assessment Guide | Hardening Guide ----|---|---|--- -Kubernetes v1.17, v1.18, & v1.19 | CIS v1.5 | [Link]({{}}/k3s/latest/en/security/self_assessment/) | [Link]({{}}/k3s/latest/en/security/hardening_guide/) - - -# Rancher with SELinux - -[Security-Enhanced Linux (SELinux)](https://en.wikipedia.org/wiki/Security-Enhanced_Linux) is a security enhancement to Linux. After being historically used by government agencies, SELinux is now industry standard and is enabled by default on CentOS 7 and 8. - -To use Rancher with SELinux, we recommend installing the `rancher-selinux` RPM according to the instructions on [this page.]({{}}/rancher/v2.6/en/security/selinux/#installing-the-rancher-selinux-rpm) diff --git a/content/rancher/v2.6/en/troubleshooting/dns/_index.md b/content/rancher/v2.6/en/troubleshooting/dns/_index.md index e9a181fc4e8..a1292df3e32 100644 --- a/content/rancher/v2.6/en/troubleshooting/dns/_index.md +++ b/content/rancher/v2.6/en/troubleshooting/dns/_index.md @@ -158,7 +158,7 @@ kubectl run -i --restart=Never --rm test-${RANDOM} --image=ubuntu --overrides='{ Enabling query logging can be done by enabling the [log plugin](https://coredns.io/plugins/log/) in the Corefile configuration in the configmap `coredns`. You can do so by using `kubectl -n kube-system edit configmap coredns` or use the command below to replace the configuration in place: ``` -kubectl get configmap -n kube-system coredns -o json | kubectl get configmap -n kube-system coredns -o json | sed -e 's_loadbalance_log\\n loadbalance_g' | kubectl apply -f - +kubectl get configmap -n kube-system coredns -o json | sed -e 's_loadbalance_log\\n loadbalance_g' | kubectl apply -f - ``` All queries will now be logged and can be checked using the command in [Check CoreDNS logging](#check-coredns-logging). diff --git a/content/rancher/v2.6/en/troubleshooting/expired-webhook-certificates/_index.md b/content/rancher/v2.6/en/troubleshooting/expired-webhook-certificates/_index.md index 99979410992..59a7bfbcc1f 100644 --- a/content/rancher/v2.6/en/troubleshooting/expired-webhook-certificates/_index.md +++ b/content/rancher/v2.6/en/troubleshooting/expired-webhook-certificates/_index.md @@ -10,6 +10,7 @@ Rancher will advise the community once there is a permanent solution in place fo ##### 1. Users with cluster access, run the following commands: ``` kubectl delete secret -n cattle-system cattle-webhook-tls +kubectl delete mutatingwebhookconfigurations.admissionregistration.k8s.io --ignore-not-found=true rancher.cattle.io kubectl delete pod -n cattle-system -l app=rancher-webhook ``` @@ -17,6 +18,8 @@ kubectl delete pod -n cattle-system -l app=rancher-webhook 1. Delete the `cattle-webhook-tls` secret in the `cattle-system` namespace in the local cluster. -1. Delete the `rancher-webhook` pod in the `cattle-system` namespace in the local cluster. +2. Delete the `rancher.cattle.io` mutating webhook -**Note:** The webhook certificate expiration issue is not specific to `cattle-webhook-tls` as listed in the examples. You will fill in your expired certificate secret accordingly. \ No newline at end of file +3. Delete the `rancher-webhook` pod in the `cattle-system` namespace in the local cluster. + +**Note:** The webhook certificate expiration issue is not specific to `cattle-webhook-tls` as listed in the examples. You will fill in your expired certificate secret accordingly. diff --git a/content/rancher/v2.6/en/troubleshooting/kubernetes-components/controlplane/_index.md b/content/rancher/v2.6/en/troubleshooting/kubernetes-components/controlplane/_index.md index 201c6e30aaf..e27339cb8b5 100644 --- a/content/rancher/v2.6/en/troubleshooting/kubernetes-components/controlplane/_index.md +++ b/content/rancher/v2.6/en/troubleshooting/kubernetes-components/controlplane/_index.md @@ -37,4 +37,12 @@ The logging of the containers can contain information on what the problem could docker logs kube-apiserver docker logs kube-controller-manager docker logs kube-scheduler +``` + +# RKE2 Server Logging + +If Rancher provisions an RKE2 cluster that can't communicate with Rancher, you can run this command on a server node in the downstream cluster to get the RKE2 server logs: + +``` +journalctl -u rke2-server -f ``` \ No newline at end of file diff --git a/content/rancher/v2.6/en/virtualization-admin/_index.md b/content/rancher/v2.6/en/virtualization-admin/_index.md index f37ffb8b951..19c517f7c3d 100644 --- a/content/rancher/v2.6/en/virtualization-admin/_index.md +++ b/content/rancher/v2.6/en/virtualization-admin/_index.md @@ -25,6 +25,8 @@ To navigate to the Harvester cluster, click **☰ > Virtualization Management**. The [Harvester node driver](https://docs.harvesterhci.io/v0.3/rancher/node-driver/) is marked as `tech preview` on RKE and RKE2 options in Rancher. This will be the case both on the Create page and once the driver is already enabled. The node driver is available whether or not the Harvester feature flag is enabled. Note that the node driver is off by default. Users may create RKE or RKE2 clusters on Harvester only from the Cluster Management page. +Harvester allows `.ISO` images to be uploaded and displayed through the Harvester UI, but this is not supported in the Rancher UI. This is because `.ISO` images usually require additional setup that interferes with a clean deployment (without requiring user intervention), and they are not typically used in cloud environments. + Click [here]({{}}/rancher/v2.6/en/admin-settings/drivers/#node-drivers) for more information on node drivers in Rancher. ### Limitations diff --git a/content/rke/latest/en/config-options/_index.md b/content/rke/latest/en/config-options/_index.md index 30b0383d445..3fe0a5ab2eb 100644 --- a/content/rke/latest/en/config-options/_index.md +++ b/content/rke/latest/en/config-options/_index.md @@ -50,7 +50,10 @@ cluster_name: mycluster ### Supported Docker Versions -By default, RKE will check the installed Docker version on all hosts and fail with an error if the version is not supported by Kubernetes. The list of [supported Docker versions](https://github.com/rancher/rke/blob/master/docker/docker.go#L37-L41) are set specifically for each Kubernetes version. To override this behavior, set this option to `true`. +By default, RKE will check the installed Docker version on all hosts and fail with an error if the version is not supported by Kubernetes. The list of supported Docker versions is set specifically for each Kubernetes version in kontainer-driver-metadata depending on the RKE version used, as shown below. To override this behavior, set this option to `true`. Refer to the following: + +- For RKE v1.3.x, see this [link](https://github.com/rancher/kontainer-driver-metadata/blob/release-v2.6/rke/k8s_docker_info.go). +- For RKE v1.2.x, see this [link](https://github.com/rancher/kontainer-driver-metadata/blob/release-v2.5/rke/k8s_docker_info.go). The default value is `false`. diff --git a/content/rke/latest/en/config-options/add-ons/_index.md b/content/rke/latest/en/config-options/add-ons/_index.md index 31c819d4b00..e449e079d77 100644 --- a/content/rke/latest/en/config-options/add-ons/_index.md +++ b/content/rke/latest/en/config-options/add-ons/_index.md @@ -27,10 +27,9 @@ There are a few things worth noting: As of version v0.1.7, add-ons are split into two categories: -- **Critical add-ons:** If these add-ons fail to deploy for any reason, RKE will error out. -- **Non-critical add-ons:** If these add-ons fail to deploy, RKE will only log a warning and continue deploying any other add-ons. +- **Critical add-ons:** If these add-ons fail to deploy for any reason, RKE will error out. All system add-ons, such as the [network plug-in]({{}}/rke/latest/en/config-options/add-ons/network-plugins/), KubeDNS, and [ingress controllers]({{}}/rke/latest/en/config-options/add-ons/ingress-controllers/), are considered critical. -Currently, only the [network plug-in]({{}}/rke/latest/en/config-options/add-ons/network-plugins/) is considered critical. KubeDNS, [ingress controllers]({{}}/rke/latest/en/config-options/add-ons/ingress-controllers/) and [user-defined add-ons]({{}}/rke/latest/en/config-options/add-ons/user-defined-add-ons/) are considered non-critical. +- **Non-critical add-ons:** If these add-ons fail to deploy, RKE will only log a warning and continue deploying any other add-ons. [User-defined add-ons]({{}}/rke/latest/en/config-options/add-ons/user-defined-add-ons/) are considered non-critical. # Add-on Deployment Jobs diff --git a/content/rke/latest/en/config-options/cloud-providers/_index.md b/content/rke/latest/en/config-options/cloud-providers/_index.md index 45501bcf784..f9a58253e15 100644 --- a/content/rke/latest/en/config-options/cloud-providers/_index.md +++ b/content/rke/latest/en/config-options/cloud-providers/_index.md @@ -3,7 +3,7 @@ title: Cloud Providers weight: 250 --- -RKE supports the ability to set your specific [cloud provider](https://kubernetes.io/docs/concepts/cluster-administration/cloud-providers/) for your Kubernetes cluster. There are specific cloud configurations for these cloud providers. +RKE supports the ability to set your specific cloud provider for your Kubernetes cluster. There are specific cloud configurations for these cloud providers. To enable a cloud provider its name as well as any required configuration options must be provided under the `cloud_provider` directive in the cluster YML. * [AWS]({{}}/rke/latest/en/config-options/cloud-providers/aws) diff --git a/content/rke/latest/en/config-options/secrets-encryption/_index.md b/content/rke/latest/en/config-options/secrets-encryption/_index.md index 81e24ff9bf9..f2539991d7f 100644 --- a/content/rke/latest/en/config-options/secrets-encryption/_index.md +++ b/content/rke/latest/en/config-options/secrets-encryption/_index.md @@ -106,14 +106,14 @@ OPTIONS: This command will perform the following actions: - Generate a new random 32-byte encryption key -- Generate a new provider configuration with the new key as the first provider and the second key as the second provider. When the secrets are rewritten, the first key will be used to encrypt the data on the write operation, while the second key (the old key) will be used to decrypt the stored data during the the read operation +- Generate a new provider configuration with the new key as the first provider and the old key as the second provider. When the secrets are rewritten, the first key will be used to encrypt the data on the write operation, while the second key (the old key) will be used to decrypt the stored data during the the read operation - Deploy the new provider configuration to all `controlplane` nodes and restart the `kube-apiserver` - Rewrite all secrets. This process will re-encrypt all the secrets with the new key. - Update the configuration to remove the old key and restart the `kube-apiserver` ### Rotating Keys by Disabling and Re-enabling Encryption in cluster.yml -For a cluster with encryption enabled, you can rotate the encryption keys by updating `cluster.yml`. If you enable and re-enable the data encryption in the `cluster.yml`, RKE will not reuse old keys. Instead, it will generate new keys every time, yielding the same result as a key rotation with the RKE CLI. +For a cluster with encryption enabled, you can rotate the encryption keys by updating `cluster.yml`. If you disable and re-enable the data encryption in the `cluster.yml`, RKE will not reuse old keys. Instead, it will generate new keys every time, yielding the same result as a key rotation with the RKE CLI. # Custom At-Rest Data Encryption Configuration With managed configuration, RKE provides the user with a very simple way to enable and disable encryption with minimal interaction and configuration. However, it doesn't allow for any customization to the configuration. @@ -139,7 +139,7 @@ kube-api: secrets_encryption_config: enabled: true custom_config: - api_version: apiserver.config.k8s.io/v1 + apiVersion: apiserver.config.k8s.io/v1 kind: EncryptionConfiguration resources: - Providers: @@ -147,7 +147,7 @@ kube-api: Keys: - Name: key1 Secret: - Resources: + resources: - secrets - identity: {} ``` diff --git a/content/rke/latest/en/config-options/services/_index.md b/content/rke/latest/en/config-options/services/_index.md index 0266731a249..77a3a969195 100644 --- a/content/rke/latest/en/config-options/services/_index.md +++ b/content/rke/latest/en/config-options/services/_index.md @@ -6,7 +6,9 @@ weight: 230 To deploy Kubernetes, RKE deploys several core components or services in Docker containers on the nodes. Based on the roles of the node, the containers deployed may be different. -**All services support additional [custom arguments, Docker mount binds and extra environment variables]({{}}/rke/latest/en/config-options/services/services-extras/).** +>**Note:** All services support additional custom arguments, Docker mount binds, and extra environment variables. +> +>To configure advanced options for Kubernetes services such as `kubelet`, `kube-controller`, and `kube-apiserver` that are not documented below, see the [`extra_args` documentation]({{}}/rke/latest/en/config-options/services/services-extras/) for more details. | Component | Services key name in cluster.yml | |-------------------------|----------------------------------| diff --git a/content/rke/latest/en/config-options/services/services-extras/_index.md b/content/rke/latest/en/config-options/services/services-extras/_index.md index 8c86d64de56..9a6d579854a 100644 --- a/content/rke/latest/en/config-options/services/services-extras/_index.md +++ b/content/rke/latest/en/config-options/services/services-extras/_index.md @@ -34,7 +34,7 @@ Additional volume binds can be added to services using the `extra_binds` argumen services: kubelet: extra_binds: - - "/host/dev:/dev" + - "/dev:/host/dev" - "/usr/libexec/kubernetes/kubelet-plugins:/usr/libexec/kubernetes/kubelet-plugins:z" ``` diff --git a/content/rke/latest/en/os/_index.md b/content/rke/latest/en/os/_index.md index c1e1dc79db8..16d05b3b9c6 100644 --- a/content/rke/latest/en/os/_index.md +++ b/content/rke/latest/en/os/_index.md @@ -148,6 +148,19 @@ https://kubic.opensuse.org/blog/2021-02-08-MicroOS-Kubic-Rancher-RKE/ If using Red Hat Enterprise Linux, Oracle Linux or CentOS, you cannot use the `root` user as [SSH user]({{}}/rke/latest/en/config-options/nodes/#ssh-user) due to [Bugzilla 1527565](https://bugzilla.redhat.com/show_bug.cgi?id=1527565). Please follow the instructions below how to setup Docker correctly, based on the way you installed Docker on the node. +>**Note:** In RHEL 8.4, two extra services are included on the NetworkManager: `nm-cloud-setup.service` and `nm-cloud-setup.timer`. These services add a routing table that interferes with the CNI plugin's configuration. If these services are enabled, you must disable them using the command below, and then reboot the node to restore connectivity: +> +> ``` + systemctl disable nm-cloud-setup.service nm-cloud-setup.timer + reboot + ``` +> +> In addition, the default firewall settings of RHEL 8.4 prevent RKE1 pods from reaching out to Rancher to connect to the cluster agent. To allow Docker containers to reach out to the internet and connect to Rancher, make the following updates to the firewall settings: +> ``` + firewall-cmd --zone=public --add-masquerade --permanent + firewall-cmd --reload + ``` + #### Using upstream Docker If you are using upstream Docker, the package name is `docker-ce` or `docker-ee`. You can check the installed package by executing: diff --git a/layouts/_default/list.html b/layouts/_default/list.html index e370b7cb515..50583bb7514 100644 --- a/layouts/_default/list.html +++ b/layouts/_default/list.html @@ -15,11 +15,11 @@
- +
- + {{ $url := urls.Parse .Permalink }} {{ $path := split $url.Path "/" }} {{ $product := index $path 1 }} @@ -30,11 +30,6 @@ We are transitioning to versioned documentation. The v2.x docs will no longer be maintained. Rancher v2.6 docs • Rancher v2.5 docs • Rancher v2.0-2.4 docs
{{end}} - {{ if in .Dir "/pipelines" }} -
- As of Rancher v2.5, Git-based deployment pipelines are now recommended to be handled with Rancher Continuous Delivery powered by Fleet, available in the Rancher UI by clicking ☰ > Continuous Delivery. -
- {{end}} {{ if in .Dir "/multi-cluster-apps" }}
As of Rancher v2.5, multi-cluster apps are deprecated. We now recommend using Fleet for deploying apps across clusters. It is available in the Rancher UI by clicking ☰ > Continuous Delivery. @@ -42,10 +37,15 @@ {{end}} {{ if in .Dir "os/v1.x" }}
- RancherOS 1.x is currently in a maintain-only-as-essential mode. It is no longer being actively maintained at a code level other than addressing critical or security fixes. For more information about the support status of RancherOS, see this page. + RancherOS 1.x is currently in a maintain-only-as-essential mode. It is no longer being actively maintained at a code level other than addressing critical or security fixes. For more information about the support status of RancherOS, see this page.
{{end}} - + {{ if or (in .Dir "/v2.0-v2.4") (in .Dir "/v2.5") }} +
+ You are viewing the documentation for an older Rancher release. If you're looking for the documentation for the latest Rancher release, go to this page instead. +
+ {{end}} +
-
diff --git a/scripts/converters/css/style-portrait.css b/scripts/converters/css/style-portrait.css index e6bcd2303ef..ab19e788e28 100644 --- a/scripts/converters/css/style-portrait.css +++ b/scripts/converters/css/style-portrait.css @@ -238,7 +238,7 @@ h2 { font-size:1.5em; } -h3 {font-size:1.4em;} +h3 {font-size:1.2em;} h4 {font-size:1.3em; line-height:30px; } @@ -283,7 +283,7 @@ nav ul li a { nav ul li a::after {content: target-counter(attr(href url), page, decimal); float:right;margin-right:10px;} nav ul li ul {list-style-type: none; border-left-style: dashed; border-left-width: 1px; border-color: #000; margin-top:1.5em;} nav ul li ul li {margin-left:-.5em;color:#ff0000;} -nav ul li ul li a {border:none;font-family:PoppinsExtraLight;margin-top:-1.5em;} +nav ul li ul li a {border:none;font-family:PoppinsExtraLight;font-size:.75em;margin-bottom:1.8em;} nav ul li ul li a::after {font-size:.75em;} nav code {background:none;} nav a{text-decoration:none;outline:none;color:#000;} diff --git a/scripts/converters/headers/header-2.6.md b/scripts/converters/headers/header-2.6.md new file mode 100755 index 00000000000..5f930a99fb1 --- /dev/null +++ b/scripts/converters/headers/header-2.6.md @@ -0,0 +1,36 @@ +--- +title: CIS v1.6 Benchmark - Self-Assessment Guide - Rancher v2.6 +weight: 101 +--- + +### CIS v1.6 Kubernetes Benchmark - Rancher v2.6 with Kubernetes v1.18 to v1.21 + +[Click here to download a PDF version of this document](https://releases.rancher.com/documents/security/2.6/Rancher_v2-6_CIS_v1-6_Benchmark_Assessment.pdf). + +#### Overview + +This document is a companion to the [Rancher v2.6 security hardening guide]({{}}/rancher/v2.6/en/security/hardening-guides/). The hardening guide provides prescriptive guidance for hardening a production installation of Rancher, and this benchmark guide is meant to help you evaluate the level of security of the hardened cluster against each control in the benchmark. + +This guide corresponds to specific versions of the hardening guide, Rancher, CIS Benchmark and Kubernetes: + +| Hardening Guide Version | Rancher Version | CIS Benchmark Version | Kubernetes Version | +| ----------------------- | --------------- | --------------------- | ------------------- | +| Hardening Guide CIS v1.6 Benchmark | Rancher v2.6.3 | CIS v1.6 | Kubernetes v1.18, v1.19, v1.20 and v1.21 | + +Because Rancher and RKE install Kubernetes services as Docker containers, many of the control verification checks in the CIS Kubernetes Benchmark do not apply and will have a result of \`Not Applicable\`. This guide will walk through the various controls and provide updated example commands to audit compliance in Rancher created clusters. + +This document is to be used by Rancher operators, security teams, auditors and decision makers. + +For more detail about each audit, including rationales and remediations for failing tests, you can refer to the corresponding section of the CIS Kubernetes Benchmark v1.6. You can download the benchmark, after creating a free account, in [Center for Internet Security (CIS)](https://www.cisecurity.org/benchmark/kubernetes/). + +#### Testing controls methodology + +Rancher and RKE install Kubernetes services via Docker containers. Configuration is defined by arguments passed to the container at the time of initialization, not via configuration files. + +Where control audits differ from the original CIS benchmark, the audit commands specific to Rancher are provided for testing. When performing the tests, you will need access to the Docker command line on the hosts of all three RKE roles. The commands also make use of the [kubectl](https://kubernetes.io/docs/tasks/tools/) (with a valid configuration file) and [jq](https://stedolan.github.io/jq/) tools, which are required in the testing and evaluation of test results. + +> NOTE: Only `automated` tests (previously called `scored`) are covered in this guide. + +### Controls + +--- diff --git a/scripts/converters/headers/header-k3s.md b/scripts/converters/headers/header-k3s.md new file mode 100755 index 00000000000..80461cc9c3f --- /dev/null +++ b/scripts/converters/headers/header-k3s.md @@ -0,0 +1,35 @@ +--- +title: CIS Self Assessment Guide +weight: 90 +--- + +### CIS Kubernetes Benchmark v1.6 - K3s with Kubernetes v1.17 to v1.21 + +#### Overview + +This document is a companion to the [K3s security hardening guide]({{}}/k3s/latest/en/security/hardening_guide/). The hardening guide provides prescriptive guidance for hardening a production installation of K3s, and this benchmark guide is meant to help you evaluate the level of security of the hardened cluster against each control in the CIS Kubernetes Benchmark. It is to be used by K3s operators, security teams, auditors, and decision-makers. + +This guide is specific to the **v1.17**, **v1.18**, **v1.19**, **v1.20** and **v1.21** release line of K3s and the **v1.6** release of the CIS Kubernetes Benchmark. + +For more information about each control, including detailed descriptions and remediations for failing tests, you can refer to the corresponding section of the CIS Kubernetes Benchmark v1.6. You can download the benchmark, after creating a free account, in [Center for Internet Security (CIS)](https://www.cisecurity.org/benchmark/kubernetes/). + +#### Testing controls methodology + +Each control in the CIS Kubernetes Benchmark was evaluated against a K3s cluster that was configured according to the accompanying hardening guide. + +Where control audits differ from the original CIS benchmark, the audit commands specific to K3s are provided for testing. + +These are the possible results for each control: + +- **Pass** - The K3s cluster under test passed the audit outlined in the benchmark. +- **Not Applicable** - The control is not applicable to K3s because of how it is designed to operate. The remediation section will explain why this is so. +- **Warn** - The control is manual in the CIS benchmark and it depends on the cluster's use case or some other factor that must be determined by the cluster operator. These controls have been evaluated to ensure K3s does not prevent their implementation, but no further configuration or auditing of the cluster under test has been performed. + +This guide makes the assumption that K3s is running as a Systemd unit. Your installation may vary and will require you to adjust the "audit" commands to fit your scenario. + +> NOTE: Only `automated` tests (previously called `scored`) are covered in this guide. + +### Controls + +--- + diff --git a/scripts/converters/run_results_to_md.sh b/scripts/converters/run_results_to_md.sh index acca856e0b1..ae3ac700b9d 100755 --- a/scripts/converters/run_results_to_md.sh +++ b/scripts/converters/run_results_to_md.sh @@ -2,8 +2,9 @@ results=${1:?path to kube-bench json results is a required argument} test_helpers=${2:?path to kube-bench test_helpers scripts is a required argument} +header=${3:?path to header file is a required argument} [ -f ${results} ] || (echo "file:'${results}' does not exist"; exit 1) [ -d ${test_helpers} ] || (echo "dir: '${test_helpers}' not a valid directory"; exit 1) -docker run -v${results}:/source/results.json -v ${test_helpers}:/test_helpers -it --rm doc_converters:latest results_to_md +docker run -v ${results}:/source/results.json -v ${test_helpers}:/test_helpers -v ${header}:/headers/header.md -it --rm doc_converters:latest results_to_md diff --git a/scripts/converters/scripts/results_to_md.sh b/scripts/converters/scripts/results_to_md.sh index 453dcbde069..a7d5d473313 100755 --- a/scripts/converters/scripts/results_to_md.sh +++ b/scripts/converters/scripts/results_to_md.sh @@ -1,48 +1,11 @@ #!/bin/bash -#results_file="${1:-/source/results.json}" -results_file="${1:-/home/paraglade/brain/projects/cis_benchmark/clusters/cis/csr.json}" -#test_helpers="${2:-/test_helpers}" -test_helpers="${2:-/home/paraglade/brain/repos/rancher-security-scan/package/helper_scripts}" +results_file="${1:-/source/results.json}" +test_helpers="${2:-/test_helpers}" +header_file="${3:-/headers/header.md}" header() { -cat < NOTE: only scored tests are covered in this guide. - -### Controls -EOF +cat ${header_file} } get_ids() { @@ -56,7 +19,7 @@ get_id_text() { get_section_ids() { id=${1} - jq -r --arg id "${id}" '.[] | select(.id==$id) | .checks[].id' ${results_file} + jq -r --arg id "${id}" '.[] | select(.id==$id) | .checks[].id' ${results_file} | sort -V } get_section_desc() { @@ -90,12 +53,11 @@ for id in $(get_ids); do test_desc=$(echo ${result} | jq -r '.description') audit=$(echo ${result} | jq -r '.audit') audit_config=$(echo ${result} | jq -r '.audit_config') - actual_value=$(echo ${result} | jq -r '.actual_value_per_node."cis-aio-0"') + actual_value=$(echo ${result} | jq -r '.actual_value_per_node[]') type=$(echo ${result} | jq -r '.test_type') status=$(echo ${result} | jq -r '.state') remediation=$(echo ${result} | jq -r '.remediation') expected_result=$(echo ${result} | jq -r '.expected_result') -# echo "#### ${test} ${test_desc}" echo if [ "${type}" = "skip" ]; then echo "**Result:** Not Applicable" @@ -113,7 +75,7 @@ for id in $(get_ids); do if [[ ${audit} =~ ".sh" ]]; then audit_script=$(basename $(echo ${audit} | cut -d ' ' -f1)) test_helper="${test_helpers}/${audit_script}" - echo "**Audit Script:** ${audit_script}" + echo "**Audit Script:** \`${audit_script}\`" echo echo '```bash' cat ${test_helper} @@ -143,6 +105,14 @@ for id in $(get_ids); do echo '```' echo fi + if [ ! -z "${expected_result}" ]; then + echo "**Expected Result**:" + echo + echo '```console' + echo ${expected_result} + echo '```' + echo + fi if [ ! -z "${actual_value}" ] && [ "${status}" != "PASS" ] && [ "${type}" != "skip" ] && [ "${type}" != "manual" ]; then echo "**Returned Value**:" echo @@ -151,14 +121,6 @@ for id in $(get_ids); do echo '```' echo fi - if [ ! -z "${expected_result}" ]; then - echo "**Expected result**:" - echo - echo '```console' - echo ${expected_result} - echo '```' - echo - fi done done done diff --git a/static/img/rancher/cilium-logo.png b/static/img/rancher/cilium-logo.png new file mode 100644 index 00000000000..681a0b3c530 Binary files /dev/null and b/static/img/rancher/cilium-logo.png differ diff --git a/static/img/rancher/neuvector-architecture.png b/static/img/rancher/neuvector-architecture.png new file mode 100644 index 00000000000..d690a1af3b9 Binary files /dev/null and b/static/img/rancher/neuvector-architecture.png differ diff --git a/static/img/rancher/neuvector-security-containers.png b/static/img/rancher/neuvector-security-containers.png new file mode 100644 index 00000000000..1750a2444e6 Binary files /dev/null and b/static/img/rancher/neuvector-security-containers.png differ diff --git a/static/img/rancher/ranchercomponentsdiagram-2.6.svg b/static/img/rancher/ranchercomponentsdiagram-2.6.svg new file mode 100644 index 00000000000..f63b84b3e75 --- /dev/null +++ b/static/img/rancher/ranchercomponentsdiagram-2.6.svg @@ -0,0 +1,2 @@ + +
Rancher 2.x
[Not supported by viewer]
Importing Kubernetes Clusters
<b>Importing Kubernetes Clusters</b>

  • Any Kubernetes cluster                      
[Not supported by viewer]
Provisioning Kubernetes Clusters
<b>Provisioning Kubernetes Clusters</b>
Rancher
<b>Rancher</b>
Rancher Launched Kubernetes
  • Nodes Hosted in an Infrastructure Provider
    • Amazon EC2
    • DigitalOcean
    • Azure
    • vSphere
    • Any (pluggable)
[Not supported by viewer]
Rancher Agents
<b>Rancher Agents</b>
Provisioning
<b>Provisioning</b>
API Framework & Types
[Not supported by viewer]
Utilities
[Not supported by viewer]
Rancher Server
https://github.com/rancher/rancher
Rancher Server<br>https://github.com/rancher/rancher<br>
RKE
https://github.com/rancher/rke
RKE<br>https://github.com/rancher/rke<br>
docker-machine
https://github.com/rancher/machine
docker-machine<br>https://github.com/rancher/machine<br>
Rancher CLI
https://github.com/rancher/cli
Rancher CLI<br>https://github.com/rancher/cli<br>
Norman
https://github.com/rancher/norman
Norman<br>https://github.com/rancher/norman<br>
Types
https://github.com/rancher/types
Types<br>https://github.com/rancher/types<br>
cattle-cluster-agent

cattle-cluster-agent<br><br>
cattle-node-agent
cattle-node-agent<br>
Rancher Launched Kubernetes
  • Creating a Cluster with Custom Nodes
[Not supported by viewer]
Rancher Agents
<b>Rancher Agents</b>
Provisioning
<b>Provisioning</b>
RKE
https://github.com/rancher/rke
RKE<br>https://github.com/rancher/rke<br>
cattle-cluster-agent

cattle-cluster-agent<br><br>
cattle-node-agent
cattle-node-agent<br>
Hosted Kubernetes Providers
  • Google Kubernetes Engine (GKE)
  • Amazon Elastic Container Service for Kubernetes (EKS)
  • Azure Kubernetes Service (AKS)
[Not supported by viewer]
Rancher Agents
<b>Rancher Agents</b>
Provisioning
<b>Provisioning</b>
kontainer-engine
https://github.com/rancher/
kontainer-engine
[Not supported by viewer]
cattle-cluster-agent

cattle-cluster-agent<br><br>
cattle-node-agent
cattle-node-agent<br>
Rancher Agents
<b>Rancher Agents</b>
cattle-cluster-agent

cattle-cluster-agent<br><br>
cattle-node-agent
cattle-node-agent<br>
User Interface
[Not supported by viewer]
Rancher Dashboard UI
https://github.com/rancher/dashboard
Rancher Dashboard UI<br>https://github.com/rancher/dashboard<br>
Rancher API UI
https://github.com/rancher/api-ui
Rancher API UI<br>https://github.com/rancher/api-ui<br>
Catalog Applications
https://github.com/rancher/helm
Catalog Applications<br>https://github.com/rancher/helm<br> \ No newline at end of file