From 147052d41f6882aeba4546687d39a73be47e6de5 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Steenis Date: Tue, 2 Oct 2018 16:28:34 +0200 Subject: [PATCH] Reorganize HPA page based on latest releases --- .../horitzontal-pod-autoscaler/_index.md | 623 ++++++++++-------- src/img/rancher/horizontal-pod-autoscaler.jpg | Bin 0 -> 38147 bytes 2 files changed, 332 insertions(+), 291 deletions(-) create mode 100644 src/img/rancher/horizontal-pod-autoscaler.jpg diff --git a/content/rancher/v2.x/en/k8s-in-rancher/horitzontal-pod-autoscaler/_index.md b/content/rancher/v2.x/en/k8s-in-rancher/horitzontal-pod-autoscaler/_index.md index 8926a33aeb5..8b6bd0d4c78 100644 --- a/content/rancher/v2.x/en/k8s-in-rancher/horitzontal-pod-autoscaler/_index.md +++ b/content/rancher/v2.x/en/k8s-in-rancher/horitzontal-pod-autoscaler/_index.md @@ -5,6 +5,8 @@ weight: 2300 Using the Kubernetes [Horizontal Pod Autoscaler](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/) feature (HPA), you can configure your cluster to automatically scale the services it's running up or down. +>**Note:** Clusters created in Rancher v2.0.7 and higher have all the requirements needed (metrics-server and Kubernetes cluster configuration) to use Horizontal Pod Autoscaler. + ### Why Use Horizontal Pod Autoscaler? Using HPA, you can automatically scale the number of pods within a replication controller, deployment, or replica set up or down. HPA automatically scales the number of pods that are running for maximum efficiency. Factors that affect the number of pods include: @@ -20,11 +22,10 @@ HPA improves your services by: ### How HPA Works -![HPA Schema]({{< baseurl >}}/img/rancher/horizontal-pod-autoscaler.svg) +![HPA Schema]({{< baseurl >}}/img/rancher/horizontal-pod-autoscaler.jpg) HPA is implemented as a control loop, with a period controlled by the `kube-controller-manager` flags below: - Flag | Default | Description | ---------|----------|----------| `--horizontal-pod-autoscaler-sync-period` | `30s` | How often HPA audits resource/custom metrics in a deployment. @@ -36,13 +37,13 @@ For full documentation on HPA, refer to the [Kubernetes Documentation](https://k ### Horizontal Pod Autoscaler API Objects -HPA is an API resource in the Kubernetes `autoscaling` API group. The current stable version is `autoscaling/v1`, which only includes support for CPU autoscaling. To get additional support for scaling based on memory and custom metrics, use the beta version instead: `autoscaling/v2beta1`. +HPA is an API resource in the Kubernetes `autoscaling` API group. The current stable version is `autoscaling/v1`, which only includes support for CPU autoscaling. To get additional support for scaling based on memory and custom metrics, use the beta version instead: `autoscaling/v2beta1`. For more information about the HPA API object, see the [HPA GitHub Readme](https://git.k8s.io/community/contributors/design-proposals/autoscaling/horizontal-pod-autoscaler.md#horizontalpodautoscaler-object). ### kubectl Commands -You can create, manage, and delete HPAs using kubectl: +You can create, manage, and delete HPAs using kubectl: - Creating HPA @@ -98,113 +99,39 @@ Directive | Description `targetAverageValue: 100Mi` | Indicates the deployment will scale pods up when the average running pod uses more that 100Mi of memory.
-### Installation - -Before you can use HPA in your Kubernetes cluster, you must fulfill some requirements. - -#### Requirements - -Be sure that your Kubernetes cluster services are running with these flags at minimum: - -- kube-api: `requestheader-client-ca-file` -- kubelet: `read-only-port` at 10255 -- kube-controller: Optional, just needed if distinct values than default are required. - - - `horizontal-pod-autoscaler-downscale-delay: "5m0s"` - - `horizontal-pod-autoscaler-upscale-delay: "3m0s"` - - `horizontal-pod-autoscaler-sync-period: "30s"` - -For an RKE Kubernetes cluster definition, add this snippet in the `services` section. To add this snippet using the Rancher v2.0 UI, open the **Clusters** view and select **Ellipsis (...) > Edit** for the cluster in which you want to use HPA. Then, from **Cluster Options**, click **Edit as YAML**. Add the following snippet to the `services` section: - -``` -services: -... - kube-api: - extra_args: - requestheader-client-ca-file: "/etc/kubernetes/ssl/kube-ca.pem" - kube-controller: - extra_args: - horizontal-pod-autoscaler-downscale-delay: "5m0s" - horizontal-pod-autoscaler-upscale-delay: "1m0s" - horizontal-pod-autoscaler-sync-period: "30s" - kubelet: - extra_args: - read-only-port: 10255 -``` - -Once the Kubernetes cluster is configured and deployed, you can deploy metrics services. - ->**Note:** kubectl command samples in the sections that follow were tested in a cluster running Rancher v2.0.6 and Kubernetes v1.10.1. - #### Configuring HPA to Scale Using Resource Metrics -To create HPA resources based on resource metrics such as CPU and memory use, you need to deploy the `metrics-server` package in the `kube-system` namespace of your Kubernetes cluster. This deployment allows HPA to consume the `metrics.k8s.io` API. +Clusters created in Rancher v2.0.7 and higher have all the requirements needed (metrics-server and Kubernetes cluster configuration) to use Horizontal Pod Autoscaler. Run the following commands to check if metrics are available in your installation: ->**Prerequisite:** You must be running kubectl 1.8 or later. +``` +$ kubectl top nodes +NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% +node-controlplane 196m 9% 1623Mi 42% +node-etcd 80m 4% 1090Mi 28% +node-worker 64m 3% 1146Mi 29% +$ kubectl -n kube-system top pods +NAME CPU(cores) MEMORY(bytes) +canal-pgldr 18m 46Mi +canal-vhkgr 20m 45Mi +canal-x5q5v 17m 37Mi +canal-xknnz 20m 37Mi +kube-dns-7588d5b5f5-298j2 0m 22Mi +kube-dns-autoscaler-5db9bbb766-t24hw 0m 5Mi +metrics-server-97bc649d5-jxrlt 0m 12Mi +$ kubectl -n kube-system logs -l k8s-app=metrics-server +I1002 12:55:32.172841 1 heapster.go:71] /metrics-server --source=kubernetes.summary_api:https://kubernetes.default.svc?kubeletHttps=true&kubeletPort=10250&useServiceAccount=true&insecure=true +I1002 12:55:32.172994 1 heapster.go:72] Metrics Server version v0.2.1 +I1002 12:55:32.173378 1 configs.go:61] Using Kubernetes client with master "https://kubernetes.default.svc" and version +I1002 12:55:32.173401 1 configs.go:62] Using kubelet port 10250 +I1002 12:55:32.173946 1 heapster.go:128] Starting with Metric Sink +I1002 12:55:32.592703 1 serving.go:308] Generated self-signed cert (apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key) +I1002 12:55:32.925630 1 heapster.go:101] Starting Heapster API server... +[restful] 2018/10/02 12:55:32 log.go:33: [restful/swagger] listing is available at https:///swaggerapi +[restful] 2018/10/02 12:55:32 log.go:33: [restful/swagger] https:///swaggerui/ is mapped to folder /swagger-ui/ +I1002 12:55:32.928597 1 serve.go:85] Serving securely on 0.0.0.0:443 +``` -1. Connect to your Kubernetes cluster using kubectl. - -1. Clone the GitHub `metrics-server` repo: - ``` - # git clone https://github.com/kubernetes-incubator/metrics-server - ``` - -1. Install the `metrics-server` package. - ``` - # kubectl create -f metrics-server/deploy/1.8+/ - ``` - -1. Check that `metrics-server` is running properly. Check the service pod and logs in the `kube-system` namespace. - - 1. Check the service pod for a status of `running`. Enter the following command: - ``` - # kubectl get pods -n kube-system - ``` - Then check for the status of `running`. - ``` - NAME READY STATUS RESTARTS AGE - ... - metrics-server-6fbfb84cdd-t2fk9 1/1 Running 0 8h - ... - ``` - 1. Check the service logs for service availability. Enter the following command: - ``` - # kubectl -n kube-system logs metrics-server-6fbfb84cdd-t2fk9 - ``` - Then review the log to confirm that that the `metrics-server` package is running. - {{% accordion id="metrics-server-run-check" label="Metrics Server Log Output" %}} - I0723 08:09:56.193136 1 heapster.go:71] /metrics-server --source=kubernetes.summary_api:'' - I0723 08:09:56.193574 1 heapster.go:72] Metrics Server version v0.2.1 - I0723 08:09:56.194480 1 configs.go:61] Using Kubernetes client with master "https://10.43.0.1:443" and version - I0723 08:09:56.194501 1 configs.go:62] Using kubelet port 10255 - I0723 08:09:56.198612 1 heapster.go:128] Starting with Metric Sink - I0723 08:09:56.780114 1 serving.go:308] Generated self-signed cert (apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key) - I0723 08:09:57.391518 1 heapster.go:101] Starting Heapster API server... - [restful] 2018/07/23 08:09:57 log.go:33: [restful/swagger] listing is available at https:///swaggerapi - [restful] 2018/07/23 08:09:57 log.go:33: [restful/swagger] https:///swaggerui/ is mapped to folder /swagger-ui/ - I0723 08:09:57.394080 1 serve.go:85] Serving securely on 0.0.0.0:443 - {{% /accordion %}} - - -1. Check that the metrics api is accessible from kubectl. - - - If you are accessing the cluster directly, enter your Server URL in the kubectl config in the following format: `https://:6443`. - ``` - # kubectl get --raw /apis/metrics.k8s.io/v1beta1 - ``` - If the the API is working correctly, you should receive output similar to the output below. - ``` - {"kind":"APIResourceList","apiVersion":"v1","groupVersion":"metrics.k8s.io/v1beta1","resources":[{"name":"nodes","singularName":"","namespaced":false,"kind":"NodeMetrics","verbs":["get","list"]},{"name":"pods","singularName":"","namespaced":true,"kind":"PodMetrics","verbs":["get","list"]}]} - ``` - - - If you are accessing the cluster through Rancher, enter your Server URL in the kubectl config in the following format: `https:///k8s/clusters/`. Add the suffix `/k8s/clusters/` to API path. - ``` - # kubectl get --raw /k8s/clusters//apis/metrics.k8s.io/v1beta1 - ``` - If the the API is working correctly, you should receive output similar to the output below. - ``` - {"kind":"APIResourceList","apiVersion":"v1","groupVersion":"metrics.k8s.io/v1beta1","resources":[{"name":"nodes","singularName":"","namespaced":false,"kind":"NodeMetrics","verbs":["get","list"]},{"name":"pods","singularName":"","namespaced":true,"kind":"PodMetrics","verbs":["get","list"]}]} - ``` +If you have created your cluster in Rancher v2.0.6 or before, please refer to [Manual installation](#manual-installation) #### Configuring HPA to Scale Using Custom Metrics (Prometheus) @@ -293,210 +220,136 @@ For HPA to use custom metrics from Prometheus, package [k8s-prometheus-adapter]( {{% /accordion %}} -#### Assigning Additional Required Roles to Your HPA - -By default, HPA reads resource and custom metrics with the user `system:anonymous`. Assign `system:anonymous` the the `view-resource-metrics` and `view-custom-metrics` in the ClusterRole and ClusterRoleBindings manifests. These roles are used to access metrics. - -To do it, follow these steps: - -1. Configure kubectl to connect to your cluster. - -1. Copy the ClusterRole and ClusterRoleBinding manifest for the type of metrics you're using for your HPA. - {{% accordion id="cluster-role-resource-metrics" label="Resource Metrics: ApiGroups resource.metrics.k8s.io" %}} - apiVersion: rbac.authorization.k8s.io/v1 - kind: ClusterRole - metadata: - name: view-resource-metrics - rules: - - apiGroups: - - metrics.k8s.io - resources: - - pods - - nodes - verbs: - - get - - list - - watch - --- - apiVersion: rbac.authorization.k8s.io/v1 - kind: ClusterRoleBinding - metadata: - name: view-resource-metrics - roleRef: - apiGroup: rbac.authorization.k8s.io - kind: ClusterRole - name: view-resource-metrics - subjects: - - apiGroup: rbac.authorization.k8s.io - kind: User - name: system:anonymous - {{% /accordion %}} -{{% accordion id="cluster-role-custom-resources" label="Custom Metrics: ApiGroups custom.metrics.k8s.io" %}} - - ``` - apiVersion: rbac.authorization.k8s.io/v1 - kind: ClusterRole - metadata: - name: view-custom-metrics - rules: - - apiGroups: - - custom.metrics.k8s.io - resources: - - "*" - verbs: - - get - - list - - watch - --- - apiVersion: rbac.authorization.k8s.io/v1 - kind: ClusterRoleBinding - metadata: - name: view-custom-metrics - roleRef: - apiGroup: rbac.authorization.k8s.io - kind: ClusterRole - name: view-custom-metrics - subjects: - - apiGroup: rbac.authorization.k8s.io - kind: User - name: system:anonymous - ``` -{{% /accordion %}} -1. Create them in your cluster using one of the follow commands, depending on the metrics you're using. - ``` - # kubectl create -f - # kubectl create -f - ``` - ### Testing HPAs with a Service Deployment -For HPA to work correctly, service deployments should have resources request definitions for containers. Follow this hello-world example to test if HPA is working correctly. +For HPA to work correctly, service deployments should have resources request definitions for containers. Follow this hello-world example to test if HPA is working correctly. 1. Configure kubectl to connect to your Kubernetes cluster. 2. Copy the `hello-world` deployment manifest below. {{% accordion id="hello-world" label="Hello World Manifest" %}} - apiVersion: apps/v1beta2 - kind: Deployment - metadata: - labels: - app: hello-world - name: hello-world - namespace: default - spec: - replicas: 1 - selector: - matchLabels: - app: hello-world - strategy: - rollingUpdate: - maxSurge: 1 - maxUnavailable: 0 - type: RollingUpdate - template: - metadata: - labels: - app: hello-world - spec: - containers: - - image: rancher/hello-world - imagePullPolicy: Always - name: hello-world - resources: - requests: - cpu: 500m - memory: 64Mi - ports: - - containerPort: 80 - protocol: TCP - restartPolicy: Always - --- - apiVersion: v1 - kind: Service - metadata: - name: hello-world - namespace: default - spec: - ports: - - port: 80 - protocol: TCP - targetPort: 80 - selector: - app: hello-world +``` +apiVersion: apps/v1beta2 +kind: Deployment +metadata: + labels: + app: hello-world + name: hello-world + namespace: default +spec: + replicas: 1 + selector: + matchLabels: + app: hello-world + strategy: + rollingUpdate: + maxSurge: 1 + maxUnavailable: 0 + type: RollingUpdate + template: + metadata: + labels: + app: hello-world + spec: + containers: + - image: rancher/hello-world + imagePullPolicy: Always + name: hello-world + resources: + requests: + cpu: 500m + memory: 64Mi + ports: + - containerPort: 80 + protocol: TCP + restartPolicy: Always +--- +apiVersion: v1 +kind: Service +metadata: + name: hello-world + namespace: default +spec: + ports: + - port: 80 + protocol: TCP + targetPort: 80 + selector: + app: hello-world +``` {{% /accordion %}} - - 1. Deploy it to your cluster. ``` # kubectl create -f ``` -1. Copy one of the HPAs below based on the metric type you're using: - {{% accordion id="service-deployment-resource-metrics" label="Hello World HPA: Resource Metrics" %}} - apiVersion: autoscaling/v2beta1 - kind: HorizontalPodAutoscaler - metadata: - name: hello-world - namespace: default - spec: - scaleTargetRef: - apiVersion: extensions/v1beta1 - kind: Deployment - name: hello-world - minReplicas: 1 - maxReplicas: 10 - metrics: - - type: Resource - resource: - name: cpu - targetAverageUtilization: 50 - - type: Resource - resource: - name: memory - targetAverageValue: 1000Mi - {{% /accordion %}} - {{% accordion id="service-deployment-custom-metrics" label="Hello World HPA: Custom Metrics" %}} - apiVersion: autoscaling/v2beta1 - kind: HorizontalPodAutoscaler - metadata: - name: hello-world - namespace: default - spec: - scaleTargetRef: - apiVersion: extensions/v1beta1 - kind: Deployment - name: hello-world - minReplicas: 1 - maxReplicas: 10 - metrics: - - type: Resource - resource: - name: cpu - targetAverageUtilization: 50 - - type: Resource - resource: - name: memory - targetAverageValue: 100Mi - - type: Pods - pods: - metricName: cpu_system - targetAverageValue: 20m - {{% /accordion %}} +1. Copy one of the HPAs below based on the metric type you're using: +{{% accordion id="service-deployment-resource-metrics" label="Hello World HPA: Resource Metrics" %}} +``` +apiVersion: autoscaling/v2beta1 +kind: HorizontalPodAutoscaler +metadata: + name: hello-world + namespace: default +spec: + scaleTargetRef: + apiVersion: extensions/v1beta1 + kind: Deployment + name: hello-world + minReplicas: 1 + maxReplicas: 10 + metrics: + - type: Resource + resource: + name: cpu + targetAverageUtilization: 50 + - type: Resource + resource: + name: memory + targetAverageValue: 1000Mi +``` +{{% /accordion %}} +{{% accordion id="service-deployment-custom-metrics" label="Hello World HPA: Custom Metrics" %}} +``` +apiVersion: autoscaling/v2beta1 +kind: HorizontalPodAutoscaler +metadata: + name: hello-world + namespace: default +spec: + scaleTargetRef: + apiVersion: extensions/v1beta1 + kind: Deployment + name: hello-world + minReplicas: 1 + maxReplicas: 10 + metrics: + - type: Resource + resource: + name: cpu + targetAverageUtilization: 50 + - type: Resource + resource: + name: memory + targetAverageValue: 100Mi + - type: Pods + pods: + metricName: cpu_system + targetAverageValue: 20m +``` +{{% /accordion %}} 1. View the HPA info and description. Confirm that metric data is shown. {{% accordion id="hpa-info-resource-metrics" label="Resource Metrics" %}} -1. Enter the following command. +1. Enter the following commands. ``` # kubectl get hpa - ``` - You should receive the output that follows: - ``` NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE hello-world Deployment/hello-world 1253376 / 100Mi, 0% / 50% 1 10 1 6m - # kubectl describe hpa + # kubectl describe hpa Name: hello-world Namespace: default Labels: @@ -552,7 +405,7 @@ For HPA to work correctly, service deployments should have resources request def 1. Test that pod autoscaling works as intended.

**To Test Autoscaling Using Resource Metrics:** {{% accordion id="observe-upscale-2-pods-cpu" label="Upscale to 2 Pods: CPU Usage Up to Target" %}} -Use your load testing tool to to scale up to two pods based on CPU Usage. +Use your load testing tool to to scale up to two pods based on CPU Usage. 1. View your HPA. ``` @@ -671,7 +524,7 @@ Use your load testing to to scale down to 1 pod when all metrics are below targe Normal SuccessfulRescale 1s horizontal-pod-autoscaler New size: 1; reason: All metrics below target ``` {{% /accordion %}} -
+
**To Test Autoscaling Using Custom Metrics:** {{% accordion id="custom-observe-upscale-2-pods-cpu" label="Upscale to 2 Pods: CPU Usage Up to Target" %}} Use your load testing tool to upscale two pods based on CPU usage. @@ -855,6 +708,8 @@ Use your load testing tool to scale down to one pod when all metrics below targe ``` {{% /accordion %}} + + ### Conclusion Horizontal Pod Autoscaling is a great way to automate the number of pod you have deployed for maximum efficiency. You can use it to accommodate deployment scale to real service load and to meet service level agreements. @@ -863,4 +718,190 @@ By adjusting the `horizontal-pod-autoscaler-downscale-delay` and `horizontal-pod We've demonstrated how to setup an HPA based on custom metrics provided by Prometheus. We used the `cpu_system` metric as an example, but you can use other metrics that monitor service performance, like `http_request_number`, `http_response_time`, etc. ->**Note:**To facilitate HPA use, we are working to integrate metric-server as an addon on RKE cluster deployments. This feature is included in RKE v0.1.9-rc2 for testing, but is not officially supported as of yet. It would be supported at rke v0.1.9. \ No newline at end of file + +### Manual Installation + +>**Note:** This is only applicable to clusters created in versions before Rancher v2.0.7. + +Before you can use HPA in your Kubernetes cluster, you must fulfill some requirements. + +#### Requirements + +Be sure that your Kubernetes cluster services are running with these flags at minimum: + +- kube-api: `requestheader-client-ca-file` +- kubelet: `read-only-port` at 10255 +- kube-controller: Optional, just needed if distinct values than default are required. + + - `horizontal-pod-autoscaler-downscale-delay: "5m0s"` + - `horizontal-pod-autoscaler-upscale-delay: "3m0s"` + - `horizontal-pod-autoscaler-sync-period: "30s"` + +For an RKE Kubernetes cluster definition, add this snippet in the `services` section. To add this snippet using the Rancher v2.0 UI, open the **Clusters** view and select **Ellipsis (...) > Edit** for the cluster in which you want to use HPA. Then, from **Cluster Options**, click **Edit as YAML**. Add the following snippet to the `services` section: + +``` +services: +... + kube-api: + extra_args: + requestheader-client-ca-file: "/etc/kubernetes/ssl/kube-ca.pem" + kube-controller: + extra_args: + horizontal-pod-autoscaler-downscale-delay: "5m0s" + horizontal-pod-autoscaler-upscale-delay: "1m0s" + horizontal-pod-autoscaler-sync-period: "30s" + kubelet: + extra_args: + read-only-port: 10255 +``` + +Once the Kubernetes cluster is configured and deployed, you can deploy metrics services. + +>**Note:** kubectl command samples in the sections that follow were tested in a cluster running Rancher v2.0.6 and Kubernetes v1.10.1. + +#### Configuring HPA to Scale Using Resource Metrics + +To create HPA resources based on resource metrics such as CPU and memory use, you need to deploy the `metrics-server` package in the `kube-system` namespace of your Kubernetes cluster. This deployment allows HPA to consume the `metrics.k8s.io` API. + +>**Prerequisite:** You must be running kubectl 1.8 or later. + +1. Connect to your Kubernetes cluster using kubectl. + +1. Clone the GitHub `metrics-server` repo: + ``` + # git clone https://github.com/kubernetes-incubator/metrics-server + ``` + +1. Install the `metrics-server` package. + ``` + # kubectl create -f metrics-server/deploy/1.8+/ + ``` + +1. Check that `metrics-server` is running properly. Check the service pod and logs in the `kube-system` namespace. + + 1. Check the service pod for a status of `running`. Enter the following command: + ``` + # kubectl get pods -n kube-system + ``` + Then check for the status of `running`. + ``` + NAME READY STATUS RESTARTS AGE + ... + metrics-server-6fbfb84cdd-t2fk9 1/1 Running 0 8h + ... + ``` + 1. Check the service logs for service availability. Enter the following command: + ``` + # kubectl -n kube-system logs metrics-server-6fbfb84cdd-t2fk9 + ``` + Then review the log to confirm that that the `metrics-server` package is running. + {{% accordion id="metrics-server-run-check" label="Metrics Server Log Output" %}} + I0723 08:09:56.193136 1 heapster.go:71] /metrics-server --source=kubernetes.summary_api:'' + I0723 08:09:56.193574 1 heapster.go:72] Metrics Server version v0.2.1 + I0723 08:09:56.194480 1 configs.go:61] Using Kubernetes client with master "https://10.43.0.1:443" and version + I0723 08:09:56.194501 1 configs.go:62] Using kubelet port 10255 + I0723 08:09:56.198612 1 heapster.go:128] Starting with Metric Sink + I0723 08:09:56.780114 1 serving.go:308] Generated self-signed cert (apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key) + I0723 08:09:57.391518 1 heapster.go:101] Starting Heapster API server... + [restful] 2018/07/23 08:09:57 log.go:33: [restful/swagger] listing is available at https:///swaggerapi + [restful] 2018/07/23 08:09:57 log.go:33: [restful/swagger] https:///swaggerui/ is mapped to folder /swagger-ui/ + I0723 08:09:57.394080 1 serve.go:85] Serving securely on 0.0.0.0:443 + {{% /accordion %}} + + +1. Check that the metrics api is accessible from kubectl. + + + - If you are accessing the cluster through Rancher, enter your Server URL in the kubectl config in the following format: `https:///k8s/clusters/`. Add the suffix `/k8s/clusters/` to API path. + ``` + # kubectl get --raw /k8s/clusters//apis/metrics.k8s.io/v1beta1 + ``` + If the the API is working correctly, you should receive output similar to the output below. + ``` + {"kind":"APIResourceList","apiVersion":"v1","groupVersion":"metrics.k8s.io/v1beta1","resources":[{"name":"nodes","singularName":"","namespaced":false,"kind":"NodeMetrics","verbs":["get","list"]},{"name":"pods","singularName":"","namespaced":true,"kind":"PodMetrics","verbs":["get","list"]}]} + ``` + + - If you are accessing the cluster directly, enter your Server URL in the kubectl config in the following format: `https://:6443`. + ``` + # kubectl get --raw /apis/metrics.k8s.io/v1beta1 + ``` + If the the API is working correctly, you should receive output similar to the output below. + ``` + {"kind":"APIResourceList","apiVersion":"v1","groupVersion":"metrics.k8s.io/v1beta1","resources":[{"name":"nodes","singularName":"","namespaced":false,"kind":"NodeMetrics","verbs":["get","list"]},{"name":"pods","singularName":"","namespaced":true,"kind":"PodMetrics","verbs":["get","list"]}]} + ``` + +#### Assigning Additional Required Roles to Your HPA + +By default, HPA reads resource and custom metrics with the user `system:anonymous`. Assign `system:anonymous` to `view-resource-metrics` and `view-custom-metrics` in the ClusterRole and ClusterRoleBindings manifests. These roles are used to access metrics. + +To do it, follow these steps: + +1. Configure kubectl to connect to your cluster. + +1. Copy the ClusterRole and ClusterRoleBinding manifest for the type of metrics you're using for your HPA. + {{% accordion id="cluster-role-resource-metrics" label="Resource Metrics: ApiGroups resource.metrics.k8s.io" %}} + apiVersion: rbac.authorization.k8s.io/v1 + kind: ClusterRole + metadata: + name: view-resource-metrics + rules: + - apiGroups: + - metrics.k8s.io + resources: + - pods + - nodes + verbs: + - get + - list + - watch + --- + apiVersion: rbac.authorization.k8s.io/v1 + kind: ClusterRoleBinding + metadata: + name: view-resource-metrics + roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: view-resource-metrics + subjects: + - apiGroup: rbac.authorization.k8s.io + kind: User + name: system:anonymous + {{% /accordion %}} +{{% accordion id="cluster-role-custom-resources" label="Custom Metrics: ApiGroups custom.metrics.k8s.io" %}} + + ``` + apiVersion: rbac.authorization.k8s.io/v1 + kind: ClusterRole + metadata: + name: view-custom-metrics + rules: + - apiGroups: + - custom.metrics.k8s.io + resources: + - "*" + verbs: + - get + - list + - watch + --- + apiVersion: rbac.authorization.k8s.io/v1 + kind: ClusterRoleBinding + metadata: + name: view-custom-metrics + roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: view-custom-metrics + subjects: + - apiGroup: rbac.authorization.k8s.io + kind: User + name: system:anonymous + ``` +{{% /accordion %}} +1. Create them in your cluster using one of the follow commands, depending on the metrics you're using. + ``` + # kubectl create -f + # kubectl create -f + ``` + diff --git a/src/img/rancher/horizontal-pod-autoscaler.jpg b/src/img/rancher/horizontal-pod-autoscaler.jpg new file mode 100644 index 0000000000000000000000000000000000000000..e39eae1bff88b600818c796117d9a29c949a30f4 GIT binary patch literal 38147 zcmeFZ1ymf%7B)J#%ituqd+^}y?(XhRaDuzLyL*7(5Zpbu1a}D#AS96Rh#fz<=gR-D z_tttnGkdDOt+jVm&CuP${lfh^09isrTm%3D0sw$KJOKC006_o*7&tgM7{mjCfPjF6 zf`^8BIFR7rVBmpBC@4TAAP^M;2NM7T3zP(8-RnOg~U*`f(W8I1iinPv$Y)>Xx6@%a%y;Vu()I4J2_z+c&Yz}(Bv zlpDH+25NpN{CHeML4Om4#@@%{en}Xu z_d^qmdHS7rMbo9{**63|oy9a++st6;hro|__(p{hYw?YJy>9@B_6t=v=h-0KTmo#F z%7nGd5Zmr=98gGisAy|3!-XH3fHCH_X-hh8e6DW@gvprt&b$UzgU>G#-x^@FGquYK zuI^ahaaAkjH`>nn!DeERisi;n_Y5ANeOK|#^4(fiz`Fj$0F(3+=CqM$(TQ&ehbrdyS#bmp;n`nbQO_qR$I>9RIY*N}+hfQwIDN)NdqGbbm zS&`Nhp@mPyVmVjKks!w8AM0=2lB8)NizN6Rhdl{v9jU6IKY;-ZJ6jdq9SeFtY%O07 zWTr_j4+}xO&ab@(0%=~j8?SD^|6%f)#zo>PF2F;kQB6({mqz&FlMX3{qS*s-*_k*azI>u=p(DWqnpaYMn`CH;4(YN5QX z<*)~FG6ud_W!QGS_vywr&TkUOROf zQt&L9^m&{YfP514fvmf}cRa|8`1r#E0J|tzDCMo{hxzX!AYFQ_2vp(%MT*t&tUISCvuTkGl-zmd}C3d=Y{gw=N7x zwK|-X@%dE;34-gD8^xQttqHG7G|udTUpar*+KScmne2BtX#XI??IcT?S62>W^mPz9 zYk@XHqVJQwI(yIt!IWplI@Ai?I-2x=- zhWVA@bP_*IJ1v*Hk#resMM**=b39^V|BEMMsbxFgAFcmQf4stZ1drAeKF#lU0w7vb zb(xRnFS;jx2mfscBvb7uiHqX~+xhn*=wpxBr`m04Xnzm?Z4Y}ks@bro+BaG5cOu`y znfp(|DPMAo$~ab$u+{3F2X^?4-`woI&Y#kWO9_kjKfBz^AtWsI$U{ME{*X@;>!?r| z&l7PrXGJACd*rOrL1YcP#&3C0+N05;IkCSpi2~s$GVWxnIReM`yYT+D1L-~%(5QePnZ59raYz0 zrS$YHjM~W6kydo=rNSirye{qm`W5v-MbDkACt1%x^x{AX3G8lU#{QeeBn+Rw4dHhT z_`?CM7aQ}PBP9Snl85i4F6xoWHV(`U)w_%T)C*oj8$ zfYjy#6s(qNvn}4d-Fg){eTqxB@fT#9!(#4Zxblke6(8WgNDx|iC3W>9b8xDx&?P;|c(1zD-H}YQL@;0Pvq@4@9D6y5+p#d;-@~ z5AaTSe5R7#0fq6panRmG2N!Jd@;S49g;lv13BD3#TM;I>1%>QtNuku9@$m417^ zm=rCFIP(qp!bFEjH4;-<2mIUX%*EiM)8wt0@{0Qsx>F@=Q$5w@WO=wi747$C=-uX& zN;~s&Wo^kNbbB-9zFbv|KrM~mHk~+~ED{@^@umFDB>Wo~(ko}L|M#2c-$?!ijD78& z{pkDODu2QLIb^zo=xIjqbD6*6f@GVqpXd1e9sB3_e>h~ezGrRb&jkiESzgAd%<*fg zLs%Q3e`}s=3IEdm9c&S1E4~ z#poWAQKoSi#vrpIx1t|O=k=4vV?MINn1cajO(!;R@ZB1@sIKn5?Agn5e?33@R7dF5a{=pvGV!}UZmz4_*V^wr zyp5T-ClGULU<<2C_G)1_e{TP`T#!hsrq`7vTj0Z3BsX2ClVEn-7O&z>%X}B{~P}# z1J55$5i%nK01%+y;ETJA zIS*$Kp&w2xfGWo6lB!aJgA?spgl0Sa99IV{(Goq&nf{5v?#ljE-9n@2Yha>d;Da4;APU`OX zN}R=3>R*Ze$sqDjQR2?*Y60aQz0|+uR1kcSE{tL&TbHGsL#eTmGxP0q-&>ukgAbn* zPj^0ks63c1Y;Rt9>z(w({mMml;c_@2k*3y=Bo7VqbU9?jkzmhMia zwl*%DYlY!&+O=Q5`$WAua`jI0b)lq(WAmF&G&)y@r`g#DI;XnTX>T&AG?z2S-yayf zuU2)sy9Y?TclhKvcX^=jX64l7l_X}ndu#FDN7dHkqus`0s^OwAw*otxoz^gK6=h~c z!Y~Jjd2v`r(yx&z&xwP-6I);HMuhvIZ4@Juz3Uf8LsiWD&EJm0POF7UY!@l2Qg))M zFfqgm!P>d3R)79SlV!GNW4%V^YD)w1Vm4~+ zN84I;>>T#<-VFDEUE>eLY6{opcX1s=-Z)fB$aArI>{EEPyAKIWIw*vHk&mS$m=tM4 zjxrQG0z;x*@>!a7WRQotl+cB|j~VPGwQ%F}?yQ$|H7`4Hu^Lz! zt>&T(oEof=m@H%*vb}23y9X@kcovTKjht3N4kMpr6kTlZe47&vBdkmf@Y7|q-~sN% zZfft}DktiPBW~^JHkxG0n~`BtTx5Uv9&4R6;s-y{Ga@|EYSF?d*MpgSuqNe8s|-Kz z;(++$j8``4;=Mz7qDG?M=ii-2xz^e34sa%5n$c^ed?RU@;` zn3+iZ6@wMp!^1SAPaQ2T8N4>Pm?<|*{pWqXiNq3!0s5az%}?L94o$%Wpp4EtcIY=2=iiEgp>;N zNJh3I9U;J8W-!BM1Twl9E~pLo%T&x5*qHFc%KrtULl9)YDdsE6Egj=T6o=&>$yD~-n|^Uz zsu|c*w)1+Thx!@qO9_R9_KRyW_H~w*O4NPLi{Mp3flQ)fxUb_JNU@pq3zLYmDeu}G z3A5W>8B2@1B41(56&6y;+yj)EXQ^9FNZofFEBc*jMj4wnaY0jxDbm7i$==Y(_mFE% zqsm8!co5teUA*;&#%eIgo_uSF#SPT}9onWUiJUp^XX2bN!4wtahDrYM8%Ha5}hzlcAb8O~l#Ho>~Bk|vWu#&?is@4&k<*eBpz-qc@ zQpvR4$Hx|Rgqc!5)%&9d&5PprV&^x!9j_EkZ%>cfo_l#0O}t;7cm4eFlXUjm)~fSQ zS0B@P*Icx>ZjY{5pLd+L4!oZ~w7Lg)t-3_Nt!A--4pXLFNJPXR5v^r{d4?o=W3xj# zt1M8Elp73RkkVdFST2y5AE2GPQ&wiKnD*WFJN0d$u_}T7BuXf1#`?;^%UDAe6wzMU zJ^{KypvaO;VR+I_xjLFGh$FpttKncygtGA(kX(V193`P=fRfcSBEYz|%uG`_#r?(O zVX|FThDBXQ3CXY))F|z|fNN5UREbvYl^0>>0foxXPK_$J#Tjd5=@PXRq?>2t@ie~= zXUw{f(L1~WVS|JY?OaZdI@UE*v{ba7Uxo^k;Ye&|Na*!L5VE$b<>(aVy@>+M+gX)} z3u9I{8Yr<0JQp3b&?8;`{-0UE$gw$IAe05RnkyvuzATlJcsvIgrz}4rj&1}aq4K_s z`{mRH*WCBarJ-=V7~-)?Z>SlaH5ffB%V{Q5R_}D0JQ@r(w_jonO0Wu7laQZ@C((Ub zp_LnxW)aV+!^k5@tB(`U3Op~cMb1!)ciPOKin6t56+d1Yx>@dJ?yL#ZR}m%gl4RYb zl(H1d-wMaokddmX@Kdy(jUi^FNw@hi;;B}f_@aFhA>ds`Jv*_%V$IX+zpeTOiP58h zWCfu}aTFF3v6)!n|D+ieV@aS(^u6dGF!4Wlb&J`>;gAHw5dQN!}W5FU8jydu*;bWHY`Q#X$P&?u3SsJ zp!u;nV(o8FU65&`>exgEpAd{e4PpSPvP2j#!%89jdL19doouT$dMAmb9zEd&%Z@*9^*R5WrW9wgbfwp|1SWD=8LRf20~)EG>9qPhwv1Qc}4EEz2ILo zJ9`hgf5LukFTlZG;q^-)rmAw~1*&F8AD1al?jQdcakbD6_)3mh(xcQe7*@gjFGX6h zw-|~i&BUhJA2Jn@SI6?JR}1%E(U3w+%a)rjYpX;8bnu`@YHR_pVi@+}G%Pf>Ij>|r> zW1wXb?vefA@QRyaKa?}AqI7fb`pV0#>dP41N{CW2Q!|T|(;4-J^5P6B7TFdq>TD4w zN7d!y5|S*@$reRIhS42G*=;r=90FWdW2Y12ja)27=#{6(^D|0Z?5nehJ~EgxXuU`B z(CldU>UlHwwmRjrv=IpcDUpOIm-xHq<4-?V?`m$#K%DPE9!z(8n>@&Ft?o4RlJC?bZQT^(3p3Jy)@h6&UIC>0(uA{Y@c z_Xqe_*?_^^ejNj&-zojdeXvBRdJsbw|$$58G3-Q1kQ7*3l!`DD*Y`>r51muS)#jX31)pZkgs&#PeRrRS(9lOb#8$ zTe=s6iUyIR4tDjN{hx&-WmPmw!Ms;MjWa{A6MS);+2~zE_m1QB9PY3l*s8wOpa9>H zKr;x>Cw=W}eDjCDR^6aWufFQlB}*AY#rP6p%2vR&jn`jgv=~xjE5;!bSNvsI**K5u zg&xMEz3W^&0QId5$rI>pbe&l|Y4w&+2TIMJUDTUeMVs%6uNwSnt^4y-DftNE49hKL(N z_oAHSyglh)9VkzK3gM@SQ`(^r=d>L)@S-_1a^@Jl-u1@PzJTwC-$ce3Vb*UxvUz{F$jr zLPFb{eEXYFPUvYVQ=oJPmDPr#DQr5MydA4{BCB0#-46_!Az%N&L#_}!93zGi&7gWr z9ifT9XkaQFBWm7@wZatJKjAaaU!L_27xamj#VeDQNh_t5vM5*-%>Jg!^O*0wo$W^; z?E{_Xg_rbNd3?#?h$0lTvmy@aA?=cNXrh)x+&1@GhD=oGD_625`2-SB;1Ak~)=ERD z_lQXIbbYPL<30bU)(Y_;_q;FN3RW?H_6ia*|JzxVH5bTFh|ll>MhZSkQNB+~NxE%% zy3b@Uc*X3IPj|3fz2>%-M^P#~i`7z!#H0X+jUA)g$$9udEfzMZ{q?$_-C^vibf zc>)9#BAG?qQTZ>3(k;;xE&`DLuelEzvb8P5ZQv?ev&Lh8MHG)Eg*=0#SWgP4Oxczj zlilhNTGjs%z$|3hkSv_B#YeCot?A~HPCo^lxt_9o5;GL6dX^)G8|#>(m5Bn2Lk5jH zx1^2X7*(B6B7`_h29zPG&tGwo-fR4-b!1NEOMhI=0mw%osx%gD+?t!yC5%US3Q7Y3 zf!Hv;Uzg!b#7zgT21~>pW5uyN(zY0^Ks>#!3Q9y#kJnK90Y1fgSu$_jzR{NMoyp^8 z{{%pwr^YS+)1z#9hK}I3yNrx(N`6FnOa-WbuNH7S*Gdawhs@kg=niaV^dsR-F zr__b|>{?EGRqSxYShxsZwGRjQ3o(1aZlQxj7>?7Sb0?f{f^|rtpb+=FPD0CifzdZ2 z0Q|!&dAqyB>w7?-D;e9fdw~1nuP$4p+!k!XmOZOPd|-|-9`Dzz2?u?oc{-)v5sq-F zW}DaJ^#ddtK%o8VZpxT-MX`|MTgZ9C)(Fsn&XDi58|!Oatakv&>67Gii0*Ry!Bv*{iUdAy6Z4Y)Gra(@Af;ZMu{E!C!wj!MTH38F8(v>t`LPuzpnR;<)|gsP%?x`I3F z!N!zUYzm*pU4FPeBidHYCUBT(u|Anr`|8>HY58JT#w;hBG4MoJ6*7-|$E_Pv=|~ka zU^el#5z1L-9DcWSv4+ftN z5tOYE22{lO^_jryGp3Chd0&tn1%wSK7596v{TTV8HlBuDBpKtJyFeWhVfS4zci_|> zN`N^7<1L*x)|l;clOHiQ2-&s;?fJckc$pPbH9cK_AU>fN(6p&gj3`FT#cN8JJ!1@RjlB0t@I-ui$ac&XeA;MW3{O>$gqCWV=`1_lNz9!jGEC;(ChVKf#Iy!lKkkWC@PJ4La4@we*YWoS+nsnE-u+)ia8{SP~EafgsphTGK(%G#}j&A3R%Zdz*Dc0E= zyTHAi9DPwp(EG$^rllgDvk1X&bA`W>GI0|d^%D_zCFyn3#S6K3`N&Q3b?w}eq9LJV zSR>J3(=P?A^$|}PeA7X?E_Dg6g~NUmmROkc7D7<4=2Ee=pKJu6$bw_My0uR^c+qvu&%&;v9;1zq?oNjHzKuV7YVns@YIF~vH#aZD=cj~t zNk_Or?KT+Pl+ZzlQ_L{!SAN`vU>Xrvl#_qjCWR9ICdRulp7@DX@6!YFJFggyPVR&s zbImIGE%{y0j8gfwNF2La8^SyLfj5ka&SCPFYcLMG@En?(I#fll`)U!^(2pXj3yr^~ zjB)@?TtdRcgw9&UBwG9=v434sI#G$FK)Ci{y{za<00TMYB`WD?K!Ni)nG(nckD^7o z=G3fZXlBxfuKhB=MezAZ+I6kuNg_rUl|tJ6NxVmUiIn2S<6V)yWzAQdsx-Ye%a>H) z90jgvPBFaur9S=34UJ5Weu}=FhQn0qIe|EPj~zEhSXyu8C9V06Va_W7N-^nA5w@eT z2*`5~3}@VlKUP+Qj9|`SLM365k|Q&ju7P1r7Fiy2fg|-W!Q!+ni-Xm!G)w5&a*1SzDaSSRsHQBjGX5%ISDBgqaVd7euRgR0@-?s&mA-R)MN+w3m4QO6c5?%cP z`M0Ev2?`oYn#z!C1#hgTr^XPT`sp{7nA*tG8kC%2o1q)zVT%`|LwAj%WP%GJk06r5 zWX$t$gHQ+=aVW&a0tv81ID&Dq(2_B}Ot~O=yxA+OT%|zLr{vT)pXYJ%@vLGt#-!PO zB3zw3hMW(?NivlDqw-e;ZAuC@Rl{XHnl+CRMaiv2S+i(PM2M*Gu#hB zTu|Qn(I9Ine+j@v7oc^GrDuopn!@{s`m%~ksjMMM{aojZRe#G+Hbu!`B4#o}IBkGz zA^ljP8eJeU!{0jkQz*!>_QQcCU8yHX2YW*mRkkA*2l{){M2g}CW6ImpG0i&Dyss&f zsgfYqsQT7YygrAX1$dJ%T)~#R$sJwlNXu&%qii}A>}nA%m`Yk~8ez;DvmQw#Ogq4M zbC5n!eRCP>J~8GZnki^DogWLAK%^7t5=xWf+Mu4VG2LFyOKUg@9WXJ8IW zj^8bPPTUw97aC-~=0sB-_M+XX-a%! zXEcq(D#<8RhONUijgBo^@Fsu8(H(#CDn?3(DWIPE)-9{`En)qE{Ge7&nmhX{0@>M8 zVHiXH>jt%!X$Bzh^uS&25S2%qU~8%Ozf$8TKa>00OA_0u_7 z(Fj?bSu2u6W4&W`H0=#P#1zYa_TZ;P$+(_16Nb{kTZHWRv2*mW$jm*UCKFRqi0K2) zGKtqNNXZNaI=utWUH2a9Gk?cOx*U~YuSUrWDvwv^8u;cYnp--O1@Ke#ZkU;7X9YLf zW+_2S?-M<>_Dd*2X3a^4hzI&uL|MHhRcGj;<6~{yWvQMoD4W9wIFS?%TSV6^mrfRp zY7=w7r7!iCa(! zl!SsBASSh^(__WUkh1J+)Vxx)l;?x_6|r0Ei_!dU(l`lp_SUZ;+feL5AH3Tn@2G6c z*HE9iUZ1PzsFrl+5>2h2pBOfHde^QyhK5q~o$4#iFm<`|F z1De{)MAr^bf*#x^BWfG8cOUN+9L%9B2nlHkk5uDBPe)4?7`((57wX0fofBKGLlPsF zU@b!@-q<8a{=E4Frbnp$ZA01%Ws+{Q7JWsl3)d`STBz%xg@Q+)z^x-B zLi1FW?g8mQ*MI{JZRh8(c;_&6(Eh3g?Jq9Q!!x*7ZRtwM>Vx#Ob4KCnv3;NU)6-3w zKrF#`w(_T7nlXW%K>nOnviF(8Qv96K5&hLwVErrc;<7-&3@*tC{Fu`v!7d39nGv82 z2BjHllmWwx_aihh!cE?D&ChWesS*Fw!(YWk1hZi2A3>KFui;sdca_be$T4qAP0?;T zrZoO?d3gAV`a5N&%4Zj~;d7p|Qb*cte=W3_ilhB-Ta2}~PfSYh=w-pkamgJ&$|tln z2KnIOS9y7{pQ>3`Mi-TZJp@+mwgZ5rqKDX~_kgrk@Z3CfBG0Y+$-ua1A96LrPST}C&iWCp3veyfYRImaK>N2X)rocC_VTzSN zT_&yS^ARz(-692nP;o9wc5tr4vRw3tiHygS36fDg=M?kAJbK4kOeTe_qU!l=j&uYo z60pim#Xcgr2-3)<+->L4e59RI;qGd;Za} z^zcM^Z$v&c#3_z*!O~&NyRr%g+c#YRgcSP@8(5_&YQ>9M;QT%~DU z=`<1{NK5XEF1y6HsZGpC91GN^54K3NEJ!5o7VLT*ja+Ni(yb{2o)DnB|KxMt!uD%gef*$yyGtldH>wV&_KZGIT+!K+jKfR@&DM_kYS zqzhOK*`&bPJl{fQLa-Vh>MN11H7(v|F^zl0Fe8AZ{^X2eVs7#E_xWmah%oK2zvGF5 zDA=N@9CK!>OCA@$^rlk-G&6525l$ByB~3u5kXVMUh>DctQ%rT*@NVyTRialk1G_e| zc~)o)nOc!j&}(0kn3R(JVq#nwS}SmL#LI1$!4Vj&Gvn!ZrXV-%_4QKRh+X0^;*M(s z^M2DVWLj2HWwA_Bmk{0cCeDLLG-s))`uII>7@eQ)xvTNNfRmUt93myM5C^laV|f&1 zsZmqXi;9pPA@92Aj#twVZmABLZF~T^A#KzWzbqz9u}ZqSTI~gc_BBv}zCp-&Owete*> zRdZaLY1x>Vpi|d{)^kv?RFklcJYq~jP@oAWzRnnR7}`N>x=1!kl9S|da=9^tM>~Dx51txC)GNO@s-zmI;~%XWpRRXegd4+%Z@a` z?I&++))$iuVJ{SBa&q0B}O8 zUMjp+1;M~xm$xp{IBYG}BM}Yn#Pp#Z4&#J(h7}j5?*Y}zO+QwbecnAl)9z{Rsynu5 z(5{Ow&sJ1#Q--458C?89sy;Cdi9Q}GIx0$lt^|(qdhk)x$jN5Y;u@osO*pd*|tVHGA4<4S9W%^JcaF@p6OzudF+c-IhdN)RFGL6ZoAj9EH&%%Z=2DA2VYi-0m-{+^oecCEkG})@NxdJeqFn0;?oXnhC0}rsq2n-+4(`Nj-|pG5u9& z1w*PyvN==PJ%ZAbos^V`UZ%EXp9`vn&A_f|h^jwZWi(_o%q4vOI&Kq(Y}%BF>2=c& z8+HZMO$nd2B0l92gzk=>n(jbm*b5~TB$#-vuLR}my2V}W5!f-WBh} zd1fEun06huJCHukDxVK{6~OcSf}q3}T+<}Nu#z2u0Sb0oV=7WLmV6-f$Wxi7{DjD}Jzl*?RcwNm zz(<}g`oi~sLHcF6(vWa2*^zNPwrU9L^4Ai!C2vFU-SDrtyP-5>X-g5zH3P&;&6O92 zcj#yc=R&qE#S^;tI?U|~j2tkb)pk#Hq@7OUaK!0aqjWVUBD&~b7MiM!8{tKfqmsUq z3y+9k+p-^3b;K5lXQEFO5@Fo(G$#qh%g;kz(&{LhQ>65djPO3aD^Q>w0s@2T)3k|* zWU--@3(ckbh6i})d(Pg*)782}3<~xUky3`66<8$A^1a5!SagT6W~BVs%SAd{t0|IN zb{8c~`y*Z*J>R+b5eq>Ll|wlz0qq&{%1AAd*Wv!b`lszsBPM6;&g-IO@ zRN|SA=BEi@uicDQ-y}}P2AibHouRlyDPkQaxs`fsw25FG$n>i2N1n&#KcT(34itQa zu{w3Zq%{%o7_zj~zkHNZ*R!SX@i>;XRHCY!g}OQY>>~^29_Cf!htn50S|~P1OrDYC z0enJ~FX^zhizlOOOKdx{C`6}AV~0hLkrS*PCGgmr33Q<2g55ADYg2;R_Um&4Y(u-W zde?1*2nMIkh|%0<110&hI~gF~mD%8Gno2eGnW3l4Lw76|BoOLy8PhUdufaG|GP#C{ z)T!WdvVBjO)dt07Tp6YE3cEdgVwR7ooP7B8E82BeKgGUrnq8&P zszOYDS)piyC}s(hSpDomS}I$#Tyv*D>0qD^5vwh-F}3*P#Mc_T>B|BCU~l}kfr=h| z-$}VjEHt*0aaab^KSq3f_FO(&X5x_PVDXcP7bkDYM6u}=_T};Ce4fng8PnHw|93gH}jqSwFw}6!i5{HXZVpOV<$44>y`U9AB?w1Av4KXZ-T|xm^VGe0ui2vFCCx zrjR%6a$9Ydq84(_MqU_kmcj>Nr}>W zrDtJLQ}RnxoWi1JB`4^(c@IPSdk9rX$f=i*Jl_tfb2RV_{hz4(-Z15g4Qef0x6NP* zUZUbZaQGW-WxBA=#;3zp!2^3*j{T}V)aP4HFn0A;$TxRKQLe)`#@<{uzH}Ioe)#yA z_7m8;@YMs6lce2KHG*63s@S)vHxh5oTfmz%SfM?=od>v#JsDr&J>^S=6X`;V9&g%< zs18!y>fcA%Fk2uz|1ghpo_IEV=ei22Mf(_>hnHi?1TCraiKggCJ)D)y39S!5#tez* z8Cp(Ds>wYdqT0*|mPV$_eTe`}Dl${kfM*bDfn5k73eHARgD43%Yy^94B6HI6#hxZ1 z+PMCFWlp|l8e%Wl4cIB3=-H*Znd~}&H3rPmw|l3JIgxVGcmUSei!{7rEN=4T)74b2?Atzm%i0nEYCp z?cp|(gA!5^aw0Ih0Iy`+^kHUb_cpk`R?0}k^l|n)CYaE5q`c?#4_7iJSvt0;mtBe2 zeKeq@d5rvf;gNm1Y`glwI?r+|bQgT@@PJ7m4x0{RNJ=)-F3Rblmoz2-V)ZVeR#4mq zO1-7(0Y{cg9|n4(k7!C2b9#`Dd=G!lB&}y8p&1}EPD68zD20zDv z7n0fZrHvyofZq=DWZbWbUIHBtY8HyEqh%ff$c~!1n+Yr;sqwr24iivXW)+=T8V006YP?@f(8A|hOtoJai*^O zvE>Mbg55xvKOqJ;5XF6utuu82LFy~Y1D;F_fV!YWaOEwn77KZchKk(Dy`Nn~05?_; zJ1n)QH#FSxZf*ZCq zcIuP8vP!u30MZ=erSs1q?mf17`Yr%}W<4TDGt!s;Y#D52F?L|85w?q8 zK!%VI<1phM_amJl`t4`ZpN`rhVkbw5HQ7jcAlZxUE_1VLZYe^{O-#|W8Cadskl>hd z4@U*&U|AN)nePF%{7s6e)FCywXT3dcQP{aOB?#fAQ&WfCF%_Ut9e!N=o^$wvoCB07 zStk5lGD5F`35-G|g)j&+$lQSu28-Y)o=GloK!RYqGXqe7zdrQgh#^^Zu@EPuFDK}P zAH~9BvuPe$pWILKFq>PjJn_Ou8U9y#U@68Uq(bYHh-P{0$#!pGQG_FW9rHHYD5C3! zVXk8N+H`%vH`v^}Auf?)k+qX}+w6rZ1-rm6Q7IB{Lpo23Gz{jgCs{f&L<6wx!5O94 z-F!glD33{fQv$A!+BRFuyr>4px>6yzMRI5ap|Lj*&SSw{>>=B^W*sLt!}Pc_^EhA< zjNaFEUGY{FSCmaW{0y%|tY%-(Zx8^MS~CsJBn$G|zt47bcOnf)A@j)32J(4GKKue``xH|;FM)z=3YRz)dVynscPPk=ua1a~cYq`ERPRn@ zf@PAuBf_-4*Ta-d=Fsdw37^Vajt%#DcewAdHRl%H;KP4MMY`d9cfIT>KZe0Zchm>A zoj40^bn%gbaVG0xf&^nEz#uRe?~>4+Xs!e^M92px3ECG&kRN)zn^pH?CSyh);@&3+ zW*@7L#b<(9TNoG{@yy{(f>68(saQ>y?~zMan0ezj&-!`pbsJAL4}d)z}qC$ zxf~*H{FoNfc@M}ZK2Z86=DvDFgLNZ{?BG9kG*$rtmjxY)CxeMcgh?`gKNeO2nsO)e zZ!}I|N{G=i4N_P>5XKNxU_=OSYELrMH9XFrXuP~>QV9a&^2a;_Y*&Da5Z_F!>o0PS z_cj4`c=ciq#@OG2z@t%)K)(6z?l>IifQKcVqLb%4X(u%$rJ`?!Q+K$7k(BZjua}db z3ByB!_Yg$bUj%$f9lsMAAF%KBUVC!n!`YUQkKbmmrc+3~+bf@};TkF!$`iM$GR?d>uVbXK$KlqB>s zq={J9H%k@CP9>%_k@o}Bg~$Tk;3AqtEjWmILZ4Yb7M?b=rb~Nt zdes|42}fl%+(!YRqky1zM!$0b!!TF~Z14=eoVlm^&tCP%atPDdF95awu6#Mq4lRTfPktY9L5 ziJ~`WfFlKDq(Zn~(;ciW2TTnW2x3%`o}va?mk=KW4JYn9p!cK{rUS%1oQSe7#<1Dx z7Ci99LQ5e$Rtvr~N-vyXod<=sDR`*xEm=1Zl>nYes8)hv-@Pi`GR^CqK{=p^3mbz0 zl$^mWz>H5svIcAU1-+9tP_2(RS-!Ou2yn-p{Ylsn9Q8lEFznF=GKrwpt!%kNTpFiD6l zXx+q#MhOf064K8+AC%R5Meaoe?BuhEL>?VusS3o-5iK!FJOW?>^yyo7k9 z$lCy97~ydm%Px64+w_=}PIy3#H8xe73)lCVxMI>m%!4(gy!A`U);rm0K=#AaSoSR^ z=X0wCb%NRv+#g;3Z8!W8313DAcChv@DF!8uqk~&iZK$@^S!#djVR1csQoj79E5vWS zI$^(c@*C)f#Bu52B&w%|X2$op)+fqwqySDrdS`I0lwM>E8%*DdGv0<>0NG70yo>Dk z+>YpUzof{(Gz+mW-RqtHa=NqRYhE^yuV2x=5noX$%8TD2S*qX~fvHP{;Gi)$YLW4kaoL z&rsi$QoW4slpQZ9o8txH<%la+JB6q4^P?x8%u%LS_94iK=SSMpy9XeXG_^bDTlw`E zrB>n3X+*(z#9-Nw?9IM#nLpo zzyb>_X~{uda*mRO70Hq_NS2&4D2RcKOAZ2(C1=SXAUO$0mLxf;2qH;DK;SJHzXgWKF7p>hirb8gnslMT)Yp3^KNr$;HzW~ zR5ii1{u5<|oINu3dAb6CQ`e2JFnW*hnrD6N3KkJ{1p%NCi1<;)JwUCi`OyWO4>`#} zVX`+ab9>iWPhJ~_LPbOc8i!@PTOYPT+2;YfSiGOb_xnC$*Q4U8fw0gYZ&&m^cRBG& z*c6bE;p;0@pF?F`Bw5i`hI~|a)R)bJlLlzF9ZluK-!7r=prwWDTt!S2nQH+k#|M-G z`e&N_E}G3Wb1nz=3LOFgsFCPC_|LsK$sgchq*~j{MADP}{pI z>|{*RV&TuHGQVK#4aexba0H~T=yY7XM~yj8fSk2b%bvxD8gLc#UL7b@fie!CA<*vt zhv2y@q<3~g<>R>d$>brb37i-`+$s?w@pQ(10~t-Sy(I*+07wwZHt3-rZ#$%~O})Yx zOE9$Nv;y0AT;6r5mqqWy%`sh5P@|H>6=F{GFn3LlsPnZWY#S~U1vrB$C*nmuh+b#n zt8C_1YO%Ql(D$YqkpIM4P0V1QDvJ`jLNjxPX6!=}+5nFtyJ`5Vo$5fQf!zJM4KSW0 z=pDqKJ>HVk9B?IrK2L?-{9y7)vW!OI?YTuXdBM7ZCuH~$#JWUL;S)4TLEh*ARt0Rd zGi-*@AC>(E@Y6#mB@*eomMjV2o37&xmrRjOU=c16zC>O5!4NEog%m>|dTt#db_BBp zEm*gOhPY<{tE8^hyJQ({-j8Ghe$@+|ejl58Zed2b?J^k2_ycnMnEk1LC!KrVGEwg1sz1zG1Qx?i(lG$GxqS*PhOzuqrBCsDap>QaI|ei&rbB-oyX&yCi*L}%nLTfQEz48- ztGHtTVu9J&ad^;w?d<9V+OPX5ZO`+3(TdtnZNCH84$hY!10HN6ZUN69N5wy@7ruGE zU_bh2B|m_pV?aHkzORV?*}Cl$;$6?`bwrFPK-7l)P0r(EKzRMT@MFMH)=$Ad0EC>)yO<8S@)b!Rc@?lIt-OAel}tSfx? zhD$%psUNoI`SH`s+2^zW_x*6*Q{R1Y9(Cf7?+M|Mv*Y(pIV=C^ne(VW={vK9+x^!N z-u-;g&v&B0(RRPUy&tN;Hj}nzPZV22xa%kKC$QNCDrdkU2Ow|aKVe-9ns;>$wgf62Jjc;-w5%T{>Z zf5}+b-{}9yz+(U(_$6AL-&cW0IUz{sXnAf6~9 z5ufHj!nyyl8Gn#5{s8{V?)HPM`v>rA@Ay|~TjBph#*9C#2oXh3yBhRchx=E>&ImfU zzJ-dDoXPbJ|8ELxKwCgipKl36;}6UPar#hl27sZ#lu#cHrT?B084UG)N>nBa_XI#= z)c#GqGeequL9``6FwzTkU6r{U5a8zqQnV-=_aRQz9PB zXTb*`Blg!pJ6X+edLx2(Fdv`G2#?As4KV?rtg^xB9a-A!jQ9SrPn#q&`twWB(J4`z?yS%X8bD=I!&JBfdV4>~F%78DnI4)Ostoh#MyqG>`_+f@5O?9Hsl@6Gmq0WuhQ zVVGHVcdJ&tj{&%mZ+PZ3SZt=B=}{P<^3teoQTp^{CSQ*QW#k~|=^Cun&P{CauwO}+ zP?0juT}7*02^3`_Z8zBD#m70z9zyi)p!x{#B_PqI3#nN~i4Z3b20)Rc`QXKs(5V_jz8>G-aZ965MMF@RfYF`qokC43|5QNa2U zb{_A5B`s-LJFD58zZx}(FBk2@Z+dc^7|X<}!xsC}tn9UImF{=I z1+nh)+uk9}Cm{+6?KDK4uD3b~QHTh|$(2?LMT97)K@soB4X7HP;ty*zD+7%R}rH7E+9XRQf&-MW5u@k?T`4%hG{RyEQm)|i-`hW6I=YNWxv?%PuC#7m7>;t zM~UeTEGK#^MVFaHHG8RD{X8;eGh`$5Jds+rnA%ZlUl$=Nl8EweGCfn4k;H|8{l}WG z?&(7&3#=O01MNmCMpFY8xXJwxI}p6{pg_03yG0;8_YtWk%7SAxB+Hh`j#_52m#0yx zNDnb-6}YYBC^T8d5z*NIr#VogshvKN^Ry=Hvs}^kC#!~s*HC7*$W?Z+%lY=;#tJXX zqgfx-K7i|%K3fRTdjaz?5(t=YEU2XHO~RsH9oi3d@noz4%`WX=i{D<3oz_)o?voo6 z!taxow^w&AgLIno1_!T~Crzg7-m>FIq3qCnK=j1lVqW{pUx2wN9nT~|x@VfN9L*X( z#B+_}Sk&(RAB9V#oZBGq4 ztz1m_?rwb3@9}Oq)n{ryWJXfw#qJId3v8{v>|BqrsS>S^xCmOeVgT?%1<|R_7MdbKG2li)Cmvja8Xx2*M)e)wsh#*jCV2WkcPDg1hEJw@D%sAl`Q&B`zxmPALNbDd zuF)|{ZRC>^N^KVdI)knr&I{YkrB$1tlfXESz&CY`4k80Vcu*1VZzty;L4h3)@zMko zWMmA~pQ|wO0aRRwMVK3SoF_{#(@Hm?BF%&I%8sdCKUZNQ0>j!B{$oH;h{mkx1Z6aY z?(^*f^Tr2<2fORDXQzF&x2won(uVa7&j`(&3c*;MN|X10B|%(`kuFQ8!fEkC`|GU5 z?Wf~KwXVvd&D!*z$@$KrdXZvW9z(*3-F5VsOjwHO@0R-^hI( z5*8U)6~y>t0-bHNE=>E$kMYdrZtEXSF3)EX1{+ax#)-zh>o*!jt!-lX|+-3#lv+xVCDUOmX@bvtm#xW9T4 zxALvf4U|LmL2D2h7*Y>GzCmzx!*n$9-pI2IJ8S%_SWG?!`?NZm>e6B9Y6=O9UWmOj zgQQi4X63#~+0W3&=hebkls1oKCY67T+PPWeGStJAg#k_B9Q;ZeClLu_f;P!UpcDJl ziYAuy$WJivP)kpG77W(Lm0Kn}QrDP?X-gT9<)RWl zhUja>dJ8*tF!N~|JVX)OtV&daR7qbz>nJcGvb){Fh-?&mWTW51%+NImk+Bj;Ww-D3 z8#Quw-sCbUU;kA3&Eo|P;lnxP2l1@s1cF1)_nNIG?SVW)j7JgkRM$lVr}6VsqKkOuCwd$M*yQO#a!clNz4H6wG3n}1(+DpTnEiMoMkK&9q26Nl{g3{4BiXiqoR3jN0FB8_R-~szYr=wv zS=NQs)5)Y+s)goC3_54eCZVcQNKd-rQetSn3(?(Z5KPH>Kwdk6j1Gz#b7!K^y;EJw z^(wYmCAq#4R#-udbGZS2>-$gkSpd=|FM6r_m?)tXpnOhdk*Z|Iu&6Ym+54!Y{#1@X~<<#+H^ezYzsg3i>=A*AyClv)l~zB8GLEW5Vq`gHLM8a%#ql=Vb`R`|dn@;<8u`ZFPJErbnTSBCz6p^hN9OQQ-Mez3 zlpV&%r(TMTMik-2AfyVW-mRei`t(;O)hh!-Xi;YLuKPm}W!E_)YF_%x-h1C<{Yr@t ze-H5}GzzHRjHKQ$8+b|n0q0q=Y z7D!;?&ehn6b7$$udd{s)rcH*##-_UJm$uJYvL5g#0pPVVV0mp)qDL83j6Ooznn~Ky zx9Rc{op4vc^n2k|%X%z;huGbjqRys5YZ0|-%Sb*rBB(NpnO?^TuMq80(^&#=Rz}g% zzw}9a`ruVz9QG%#GGJ<*)~E$qhN_s$5p#y<6;q~wR{8@@JGJ>WNGukaMMVf^A18_MW5T?ght_(|x&G<@0XlSiV zP-Fgt;i?a$$BhWp7^O7#*~69qhHRGr+aZFiB*y9IF}Wk&9K>EA5Hn79v$CCPjjFlO z7;L#i1aQyRF+opFH;Ko7Gds8o^-Elum55f|dvk*~830g}(9VWBg_^j$1Z3WX9;^3b zWEe&*dD$eF@2r&Y&UH*?jL94-6sc0qjX$7{^4dTivsg7~r@pHfX;U*)(XT#5S`cRfp$ch5The87ym{ z+ZFaUdZ$#gIG%=zm0>lJKi#NH00ld-jPxF zPAJGi!51N4RfA<@thz=cAfWd56;vXPO9`fa(= zgk=5*oLw!WD45i!=jbihLdpoia;am!alIELKB^R;wqX5Il-7HR=a+>=wI}C4 zyCLDCxT~5LPQ6KQG6}yoBoA3b3mBQg1RFNtc{VRq5pm>@>&^GF`|@1#frq5F>wBTy z%_XOem8*uV7lvh!+Yq=om1$?{xG2Vh=>qL_Ol1LhQVVKOG>uaEtESIxn<9U3k-Kn*{wy-~C&OR>!z;axBC7*xSNV<-@oXC+ojLQ0+>=Q^%Tizz~ ze*kpnk~d@+xs%ui@Hn9A_n?vT2~6psk!-Tm`KqBJL4&MuHAv7~l9AcB*O|x(B}Z+T zFLI_9vrNbTo8Xf=GlZy`#X{B+Oyj8VHW*)6r-Y&4KUU;LYZon5wpFJf`VRnhICH>@ z7;%&CYV_~V=rgg?tZz9i_%S z96^G(HMPMP`G{XWJDmoSPISAfwtgfY=J!@e{gczm6VJRFFM2dcJ*WoZ|KT9QD8oR1yDjTKxw>{0q%xSif1(E z+6?7<_28G-K=ZTvf-mVO?Z%>&SFw@ev-G%l_>1dZ2Ln1I|A*n?wiD^<5K98H+f3y+ z4<27QGeJ>*1xwo^O$lJyK6$IlJX18Zn#zIZgTzMVDlH5mYQF8n~3pTw3R~`*ue(3Cb$c?p}V4&n705 z+ugj*HoEBEe&kTHhlPQlO;sZ-u5`R7q&djlZ}@#}pNfP?Z#&7EHdzm9N)Ks~3AF50 zatBXVg}+a8Vsql$EO;UNOBa!Da*zHz@EAnKZ?|kbWo;t;7U$?R{aK^ zT$FymW?&FNnh3yWp}UAJ95D~pOVkZzA8+sAex|!eI663>#o&`~Cz_txgC}6 zdEud^vfet2SNmZqmE3NJRf;R0?9&3$~O$bwmDcgPeM8y}CZijb-5VIwLXCYCgn=c#jO0$CC-onzIOzM&XJoQ>M ziik~1DOF|+G!VotP{GOoB!l^=rH;}aTU;#aA##kf$dIzJWpJ2wl-Yv8ns+ku$FPM0 zeT`8{h3i9-t#C`DN`N{EpI!qC25M>xgZrWO@9WS1xNNBxp`eu= zL#+=PmZ7C7+{gwgauRP$ibJCKR0Kh7~|QhK#~UU+i5smkWsNW5sQ=$3X>6csuJg#;SGfv zgXhSa+)i}N6^T1oYO7Q+0i^6n<}4do6;<~ySo0NogoqGKKhyyUkW+DHO4Gq>*w?h6 z5$}oHd(0>Yc-rzr$aP3DV-LIOui~4JGBWsn`hv8T3PDaq2Jj%yYW}vh3g6imFx>Ls zS2_EXT0w;nX$7mwd8X{VL~MnISUcE@m=k?llnw84PyBn?~lR)SLZY#_YJNkbxoD0d;8piTg)-Wr25JR0NY_fskdnw>xK@_o&||mPu|P%MKnn z{tyKnIW97h01%e~PG0Ttp!dQ$A^Ijak+&TcgU-eImwb7fvm^K1c5Yrcyz#(dwD;CsaGWB&u@vkGO+EODa59m&*gDUR&jF2b$6WD`g4xg~zO^sBWEwhF&9 zhPPcr^-fQ`ulixK6RzOG#aq8ey9QyjbmHh)tV*)IuOf)`f0b@}N)EI8Zxp{Rp z+TjT0(YjQ9MQht`kxJti9Hn~l&#!O^W@@C40drS`KiN#m!<0PUP#goimJSAszccN4 zD|3(MK1lArA(8@&&9H4_w9=I`v5v23#xp8`EIwLB5ha|1sF)g>AF`F(A0_i{rJ-*a zr8p6M+n% z-z)w32f}N)s!8$v{<+i~7>F-V%M-As4qid5fld`*1txIiB1lzhs z9l~g@mPm~bPwf^r^{bc^>L*$j2|FtixqU7{v!|K? z451C_kYNI3%MQWm;zNe7u;W3wQkkc22sjSObHPO@#5KniID5$`fUgT0!eUr*F|@&7 z2$LS5Z^4yG9=627$BMB+YT(I;_Z}wCS8Wp?_L530`LIiG@MxG1c&nf$+Q=5gUO*Q$ zRY+fxdr@!T0@4m~2pI$92FXtWBOIivj8N>U-(olQQc*#Qi^sZlGYlq-7VeHQ&OUu$ zBQsQyOwaJ)&Y3AYx!kU{G_GDOR(XIQ-V^wkPDLdW#s|xx+C=EX8 z@@5XoDiyI7>%PYPll)t9x7;l2=0bTm#y4{cq!8byN3@?Kc6mP=j>-`)Zwx;ytY~{j zRIG7Mu~jn^cBT$WNp*hL6iL#kMCHgWs{t=y7o3E5D{t0M1}INalS}z9f=LZoCRlj* zgIK4)Blu!ahl>Gs!0`PlO{Ch|yDIPUkT0~#qKW|DrQBP7x#geX)eC;D0^mQ)U;v0xei%b@)*S59;h5BeddSALIh;O$Zh!}(@hB}?-$?%LFtp-!`( zL0QNv`vAG9>Da|XDgkN#ZuPO zHbr4D=S_$Z1$Dm&s_+i7Nq9_UD`kKR3c<8eiop@UL)r%3Oxg(4oWItEJlHqx+ z;GB0yz3ULSGe)G8iehlvUa&$EZt6iRJS5epx=WU(2aRTo`Y- z48T>b2~G`rYZ-zHvdiC+%nXS3Sb=>7U?Df-^TZw{SmV>O@O+N8S24a)7D?>ufyVVM1L*97HC7e^Iut!A(_uGjO9F!p${p zT@gqRl&i<&X5D@DK9tt|S-O;D6I-r@ZK literal 0 HcmV?d00001