public/rancher-docs
1
0
Fork 0
mirror of https://github.com/rancher/rancher-docs.git synced 2026-09-28 05:59:03 +00:00
Code Issues Packages Projects Releases Wiki Activity

edits

Browse Source
This commit is contained in:
Mark Bishop
2018-08-16 10:29:10 -07:00
parent 097e585f9d
commit e2fb2b47d6
1 changed files with 269 additions and 236 deletions
Download Patch File Download Diff File Expand all files Collapse all files
content/rancher/v2.x/en/k8s-in-rancher/horitzontal-pod-autoscaler/_index.md
+269 -236
View File
@@ -4,53 +4,77 @@ weight: 2300
draft: true draft: true
--- ---
--- Using the Kubernetes [Horizontal Pod Autoscaler](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/) feature (HPA), you can configure your cluster to automatically scale its running services up or down.
### Introduction ### Why Use Horizontal Pod Autoscaler?
Some of the nicer features on k8s is the ability to code and configure autoscale on your running services. This feature is called Horitzontal Pod Autoscaler (hpa) on k8s clusters. Using HPA, in real time, you can automatically scale deployments up or down based on:
### Why use HPA - The cluster hardware resources in use.
- Custom metrics.
Using hpa, you can achieve up/down autoscale in your deployments, based on resources use and/or custom metrics, to accomodate deployments scale to real time load of your services. HPA improves to your services by:
HPA produce 2 direct improvements to your services, - Releasing hardware resources that would otherwise be wasted by an excessive number of pods.
1. Use compute and memory resources when are needed, releasing them if not required. - Increase/decrease performance as needed to accomplish SLA.
2. Increase/decrease performance as needed to accomplish SLA.
### How HPA works ### How HPA Works
HPA automatically scales the number of pods (defined minimum and maximum number of pods) in a replication controller, deployment or replica set, based on observed CPU/memory utilization (resource metrics) or based on custom metrics provided by third party metrics application like prometheus, datadog, etc...(custom metrics). Within a replication controller, deployment, or replica set, HPA automatically scales the number of pods that are running for maximum efficiency. Factors that affect the number of pods include:
HPA is implemented as a control loop, with a periods controlled by the k8s controller manager flags: - A minimum and maximum number of pods allowed to run, as defined by the user.
- `--horizontal-pod-autoscaler-sync-period`: how often hpa check for metrics (default value 30s).
- `--horizontal-pod-autoscaler-downscale-delay`: how long hpa has to wait before another downscale operation can be performed after the current one has completed (default value 5m0s). - Observed CPU/memory use, as reported in resource metrics.
- `--horizontal-pod-autoscaler-upscale-delay`: how long hpa has to wait before another upscale operation can be performed after the current one has completed (default value 3m0s).
- Custom metrics provided by third-party metrics application like Prometheus, Datadog, etc.
HPA is implemented as a control loop, with a period controlled by the Kubnernetes controller manager flags. Flags includes:
- `--horizontal-pod-autoscaler-sync-period`
How often HPA audits resource/custom metrics in a deployment (default value: 30s).
- `--horizontal-pod-autoscaler-downscale-delay`
Following completion of a downscale operation, how long HPA must wait before launching another downscale operations (default value 5m0s).
- `--horizontal-pod-autoscaler-upscale-delay`
Following completion of an upscale operation, how long HPA must wait before launching another upscale operation (default value 3m0s).
<img src="img/horizontal-pod-autoscaler.svg" width="800" alt="HPA schema"> <img src="img/horizontal-pod-autoscaler.svg" width="800" alt="HPA schema">
More info at [horizontal-pod-autoscale](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/) For full documentation on HPA, refer to the [Kubernetes Documentation](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/).
### HPA definition ### Horizontal Pod Autoscaler Definition
HPA is an API resource in the Kubernetes `autoscaling` API group. Current stable version is `autoscaling/v1`, which only includes support for CPU autoscaling. To get additional support for scaling on memory and custom metrics, beta vesion should be used `autoscaling/v2beta1`. HPA is an API resource in the Kubernetes `autoscaling` API group. The current stable version is `autoscaling/v1`, which only includes support for CPU autoscaling. To get additional support for scaling based on memory and custom metrics, use the beta version instead: `autoscaling/v2beta1`.
[More info about hpa API object](https://git.k8s.io/community/contributors/design-proposals/autoscaling/horizontal-pod-autoscaler.md#horizontalpodautoscaler-object) For more information about the HPA API object, see the [HPA GitHub Readme](https://git.k8s.io/community/contributors/design-proposals/autoscaling/horizontal-pod-autoscaler.md#horizontalpodautoscaler-object).
HPA is supported in a standard way by kubectl. It can be created, managed and deleted using kubectl: You can create, manage, and delete HPAs using kubectl:
- Creating HPA
- Creating hpa
- With manifest: `kubectl create -f <HPA_MANIFEST>` - With manifest: `kubectl create -f <HPA_MANIFEST>`
- Without manifest (Just support CPU): `kubectl autoscale deployment hello-world --min=2 --max=5 --cpu-percent=50` - Without manifest (Just support CPU): `kubectl autoscale deployment hello-world --min=2 --max=5 --cpu-percent=50`
- Getting hpa info
- Getting HPA info
- Basic: `kubectl get hpa hello-world` - Basic: `kubectl get hpa hello-world`
- Detailed description: `kubectl describe hpa hello-world` - Detailed description: `kubectl describe hpa hello-world`
- Deleting hpa
- Deleting HPA
- `kubectl delete hpa hello-world` - `kubectl delete hpa hello-world`
HPA manifest definition example ### HPA Manifest Definition Example
``` The following snippet demonstrates use of different directives in an HPA manifest. See the list below the sample to understand the purpose of each directive.
```yml
apiVersion: autoscaling/v2beta1 apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler kind: HorizontalPodAutoscaler
metadata: metadata:
@@ -73,21 +97,24 @@ spec:
targetAverageValue: 100Mi targetAverageValue: 100Mi
``` ```
- Using `autoscaling/v2beta1` version to use cpu and memory metrics
- Controlling autoscale of `hello-world` deployment Directive | Description
- Defined minimum number of replicas of 1 ---------|----------|
- Defined maximum number of replicas of 10 `apiVersion: autoscaling/v2beta1` | The version of the Kubernetes `autoscaling` API group in use. This example manifest uses the beta version, so scaling by CPU and memory is enabled. |
- Scaling up when: `name: hello-world` | Indicates that HPA is performing autoscaling for the `hello-word` deployment. |
- cpu use is more that 50% `minReplicas: 1` | Indicates that the minimum number of replicas running can't go below 1. |
- Memory use more than 100Mi `maxReplicas: 10` | Indicates the maximum number of replicas in the deployment can't go above 10.
`targetAverageUtilization: 50` | Indicates the deployment will scale pods up when the average running pod uses more than 50% of its requested CPU.
`targetAverageValue: 100Mi` | Indicates the deployment will scale pods up when the average running pod uses more that 100Mi of memory.
<br/>
### Installation ### Installation
Before hpa could be used at your k8s cluster, some elements have to be installed and configured in your system. Before you can use HPA in your Kubernetes cluster, you must fulfill some requirements.
#### Requirements #### Requirements
Be sure that your k8s cluster services are running at least with these flags: Be sure that your Kubernetes cluster services are running with these flags at minimum:
- kube-api: `requestheader-client-ca-file` - kube-api: `requestheader-client-ca-file`
- kubelet: `read-only-port` at 10255 - kubelet: `read-only-port` at 10255
@@ -96,7 +123,7 @@ Be sure that your k8s cluster services are running at least with these flags:
- `horizontal-pod-autoscaler-upscale-delay: "3m0s"` - `horizontal-pod-autoscaler-upscale-delay: "3m0s"`
- `horizontal-pod-autoscaler-sync-period: "30s"` - `horizontal-pod-autoscaler-sync-period: "30s"`
For RKE k8s cluster definition, be sure you add these lines at services section. To do it at Rancher v2.0.X ui, open "Cluster options" - "Edit as YAML" and add these definition: For an RKE Kubernetes cluster definition, add this snippet in the `services` section. To add this snippet using the Rancher v2.0 UI, open the **Clusters** view and select **Ellipsis (...) > Edit**. Then, from **Cluster Options**, click **Edit as YAML**. Add the following snippet to the `services` section:
``` ```
services: services:
@@ -114,27 +141,29 @@ services:
read-only-port: 10255 read-only-port: 10255
``` ```
Once k8s cluster is configured and deployed properly, is needed to deploy metrics service. Once the Kubernetes cluster is configured and deployed, you can deploy metrics services.
Note: For deploy and test examples, Rancher v2.0.6 and k8s v1.10.1 cluster are being used. >**Note:** Code samples in the sections that follow were tested in a cluster running Rancher v2.0.6 and Kubernetes v1.10.1.
#### Resource metrics #### Resource Metrics
In order to create horizontal pod autoscaler resources based on resource metrics (e.g. pod CPU/memory usage), you will need to deploy the `metrics-server` package in the `kube-system` namespace of k8s cluster, which will enable HPA to consume the `metrics.k8s.io` API. To create HPA resources based on resource metrics such as CPU and memory use, you need to deploy the `metrics-server` package in the `kube-system` namespace of your Kubernetes cluster. This deployment allows HPA to consume the `metrics.k8s.io` API.
To do it, follow these steps:
- Configure kubectl to connect proper k8s cluster. >**Prerequisite:** You must be running kubectl 1.8 or later.
- Clone github `metrics-server` repo:
1. Connect to your Kubernetes cluster using kubectl.
1. Clone the GitHub `metrics-server` repo:
``` ```
# git clone https://github.com/kubernetes-incubator/metrics-server # git clone https://github.com/kubernetes-incubator/metrics-server
``` ```
- Install `metrics-server` package (supossed that k8s is up to version 1.8): 1. Install the `metrics-server` package.
``` ```
# kubectl create -f metrics-server/deploy/1.8+/ # kubectl create -f metrics-server/deploy/1.8+/
``` ```
- Check that `metrics-server` is running properly. Check service pod and logs at namespace `kube-system` 1. Check that `metrics-server` is running properly. Check the service pod and logs at namespace `kube-system` by running the command below.
``` ```
# kubectl get pods -n kube-system # kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE NAME READY STATUS RESTARTS AGE
@@ -142,6 +171,7 @@ To do it, follow these steps:
metrics-server-6fbfb84cdd-t2fk9 1/1 Running 0 8h metrics-server-6fbfb84cdd-t2fk9 1/1 Running 0 8h
... ...
``` ```
If metric-server is running properly, you'll receive output similar to the output below.
``` ```
# kubectl -n kube-system logs metrics-server-6fbfb84cdd-t2fk9 # kubectl -n kube-system logs metrics-server-6fbfb84cdd-t2fk9
I0723 08:09:56.193136 1 heapster.go:71] /metrics-server --source=kubernetes.summary_api:'' I0723 08:09:56.193136 1 heapster.go:71] /metrics-server --source=kubernetes.summary_api:''
@@ -156,50 +186,52 @@ To do it, follow these steps:
I0723 08:09:57.394080 1 serve.go:85] Serving securely on 0.0.0.0:443 I0723 08:09:57.394080 1 serve.go:85] Serving securely on 0.0.0.0:443
``` ```
- Check that metrics api is accesible from kubectl 1. Check that the metrics api is accessible from kubectl.
- If you are accessing directly to k8s cluster, server url at kubectl config like 'https://<K8s_URL>:6443' - If you are accessing the cluster directly, enter your Server URL in the kubectl config in the following format: `https://<Kubernetes_URL>:6443`.
``` ```
# kubectl get --raw /apis/metrics.k8s.io/v1beta1 # kubectl get --raw /apis/metrics.k8s.io/v1beta1
{"kind":"APIResourceList","apiVersion":"v1","groupVersion":"metrics.k8s.io/v1beta1","resources":[{"name":"nodes","singularName":"","namespaced":false,"kind":"NodeMetrics","verbs":["get","list"]},{"name":"pods","singularName":"","namespaced":true,"kind":"PodMetrics","verbs":["get","list"]}]} {"kind":"APIResourceList","apiVersion":"v1","groupVersion":"metrics.k8s.io/v1beta1","resources":[{"name":"nodes","singularName":"","namespaced":false,"kind":"NodeMetrics","verbs":["get","list"]},{"name":"pods","singularName":"","namespaced":true,"kind":"PodMetrics","verbs":["get","list"]}]}
``` ```
- If you are accessing to k8s cluster throught rancher, server url at kubectl config like `https://<RANCHER_URL>/k8s/clusters/<CLUSTER_ID>` You need to add prefix `/k8s/clusters/<CLUSTER_ID>` to api path - If you are accessing the cluster through Rancher, enter your Server URL in the kubectl config in the following format: `https://<RANCHER_URL>/k8s/clusters/<CLUSTER_ID>`. Add the suffix `/k8s/clusters/<CLUSTER_ID>` to API path.
``` ```
# kubectl get --raw /k8s/clusters/<CLUSTER_ID>/apis/metrics.k8s.io/v1beta1 # kubectl get --raw /k8s/clusters/<CLUSTER_ID>/apis/metrics.k8s.io/v1beta1
{"kind":"APIResourceList","apiVersion":"v1","groupVersion":"metrics.k8s.io/v1beta1","resources":[{"name":"nodes","singularName":"","namespaced":false,"kind":"NodeMetrics","verbs":["get","list"]},{"name":"pods","singularName":"","namespaced":true,"kind":"PodMetrics","verbs":["get","list"]}]} {"kind":"APIResourceList","apiVersion":"v1","groupVersion":"metrics.k8s.io/v1beta1","resources":[{"name":"nodes","singularName":"","namespaced":false,"kind":"NodeMetrics","verbs":["get","list"]},{"name":"pods","singularName":"","namespaced":true,"kind":"PodMetrics","verbs":["get","list"]}]}
``` ```
#### Custom metrics (prometheus) #### Custom Metrics (Prometheus)
Besides acting on resource metrics, HPA can be configured to autoscale based on custom metrics provided by a third party. The most important use case is to be able to autoscale based on application-level metrics (e.g. HTTP requests per second). HPA uses the `custom.metrics.k8s.io` API to consume these metrics. This API is enabled by deploying a custom metrics adapter corresponding to the metrics collection solution. You can configure HPA to autoscale based on custom metrics provided by third-party software. The most common use case for autoscaling using third-party software is based on application-level metrics (e.g. HTTP requests per second). HPA uses the `custom.metrics.k8s.io` API to consume these metrics. This API is enabled by deploying a custom metrics adapter corresponding to the metrics collection solution.
We are gonna use [prometheus](https://prometheus.io/) for the example. We are assuming that prometheus is deployed at k8s cluster, getting proper metrics from pods, nodes, namespaces,.... We'll use prometehus url, http://prometheus.mycompany.io exposed at port 80 For this example, we are going to use [Prometheus](https://prometheus.io/). We are beggining with the following assumptions:
Prometheus is available for deploy in rancher v2.0 on catalog. Deploy it from rancher catalog if it isn't alrady running on your k8s cluster. - Prometheus is deployed in the cluster.
- Prometheus is configured correctly and collecting proper metrics from pods, nodes, namespaces, etc.
- Prometheus is exposed at the following URL and port: `http://prometheus.mycompany.io:80`
If hpa wants to use custom metrics from Prometheus, package [k8s-prometheus-adapter](https://github.com/DirectXMan12/k8s-prometheus-adapter) is needed at `kube-system` namespace on k8s cluster. Just to facilitate `k8s-prometheus-adapter` installation, we are gonna to use helm chart available at [banzai-charts](https://github.com/banzaicloud/banzai-charts) Prometheus is available for deployment in the Rancher v2.0 catalog. Deploy it from Rancher catalog if it isn't already running in your cluster.
To do it, follow these steps: If HPA wants to use custom metrics from Prometheus, package [k8s-prometheus-adapter](https://github.com/DirectXMan12/k8s-prometheus-adapter) is needed at `kube-system` namespace on k8s cluster. Just to facilitate `k8s-prometheus-adapter` installation, we are going to use Helm chart available at [banzai-charts](https://github.com/banzaicloud/banzai-charts).
- Init helm at k8s cluster 1. Initialize Helm in your cluster
``` ```
# kubectl -n kube-system create serviceaccount tiller # kubectl -n kube-system create serviceaccount tiller
kubectl create clusterrolebinding tiller --clusterrole cluster-admin --serviceaccount=kube-system:tiller kubectl create clusterrolebinding tiller --clusterrole cluster-admin --serviceaccount=kube-system:tiller
helm init --service-account tiller helm init --service-account tiller
``` ```
- Clone github `banzai-charts` repo: 1. Clone the `banzai-charts` Helm chart from GitHub:
``` ```
# git clone https://github.com/banzaicloud/banzai-charts # git clone https://github.com/banzaicloud/banzai-charts
``` ```
- Install `prometheus-adapter` char specifying prometheus url and port 1. Install `prometheus-adapter` char specifying prometheus url and port.
``` ```
# helm install --name prometheus-adapter banzai-charts/prometheus-adapter --set prometheus.url="http://prometheus.mycompany.io",prometheus.port="80" --namespace kube-system # helm install --name prometheus-adapter banzai-charts/prometheus-adapter --set prometheus.url="http://prometheus.mycompany.io",prometheus.port="80" --namespace kube-system
``` ```
- Check that `prometheus-adapter` is running properly. Check service pod and logs at namespace `kube-system` 1. Check that `prometheus-adapter` is running properly. Check the service pod and logs in the `kube-system` namespace.
``` ```
# kubectl get pods -n kube-system # kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE NAME READY STATUS RESTARTS AGE
@@ -223,15 +255,15 @@ To do it, follow these steps:
... ...
``` ```
- Check that metrics api is accesible from kubectl 1. Check that the metrics api is accessible from kubectl.
- Accessing directly to k8s cluster, server url at kubectl config like 'https://<K8s_URL>:6443' - If you are accessing the cluster directly, enter your Server URL in the kubectl config in the following format: `https://<Kubernetes_URL>:6443`.
``` ```
# kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1 # kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1
{"kind":"APIResourceList","apiVersion":"v1","groupVersion":"custom.metrics.k8s.io/v1beta1","resources":[{"name":"pods/fs_usage_bytes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/memory_rss","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/spec_cpu_period","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/cpu_cfs_throttled","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_io_time","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_read","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_sector_writes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/cpu_user","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/last_seen","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/tasks_state","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/spec_cpu_quota","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/start_time_seconds","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_limit_bytes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_write","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/memory_cache","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/memory_usage_bytes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/cpu_cfs_periods","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/cpu_cfs_throttled_periods","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_reads_merged","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/memory_working_set_bytes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/network_udp_usage","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_inodes_free","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_inodes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_io_time_weighted","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/memory_failures","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/memory_swap","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/spec_cpu_shares","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/spec_memory_swap_limit_bytes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/cpu_usage","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_io_current","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_writes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/memory_failcnt","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_reads","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_writes_bytes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_writes_merged","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/network_tcp_usage","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/memory_max_usage_bytes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/spec_memory_limit_bytes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/spec_memory_reservation_limit_bytes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/cpu_load_average_10s","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/cpu_system","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_reads_bytes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_sector_reads","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]}]} {"kind":"APIResourceList","apiVersion":"v1","groupVersion":"custom.metrics.k8s.io/v1beta1","resources":[{"name":"pods/fs_usage_bytes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/memory_rss","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/spec_cpu_period","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/cpu_cfs_throttled","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_io_time","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_read","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_sector_writes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/cpu_user","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/last_seen","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/tasks_state","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/spec_cpu_quota","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/start_time_seconds","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_limit_bytes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_write","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/memory_cache","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/memory_usage_bytes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/cpu_cfs_periods","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/cpu_cfs_throttled_periods","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_reads_merged","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/memory_working_set_bytes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/network_udp_usage","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_inodes_free","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_inodes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_io_time_weighted","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/memory_failures","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/memory_swap","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/spec_cpu_shares","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/spec_memory_swap_limit_bytes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/cpu_usage","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_io_current","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_writes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/memory_failcnt","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_reads","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_writes_bytes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_writes_merged","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/network_tcp_usage","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/memory_max_usage_bytes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/spec_memory_limit_bytes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/spec_memory_reservation_limit_bytes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/cpu_load_average_10s","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/cpu_system","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_reads_bytes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_sector_reads","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]}]}
``` ```
- Accessing to k8s cluster throught rancher, server url at kubectl config like `https://<RANCHER_URL>/k8s/clusters/<CLUSTER_ID>` You need to add prefix `/k8s/clusters/<CLUSTER_ID>` - If you are accessing the cluster through Rancher, enter your Server URL in the kubectl config in the following format: `https://<RANCHER_URL>/k8s/clusters/<CLUSTER_ID>`. Add the suffix `/k8s/clusters/<CLUSTER_ID>` to API path.
``` ```
# kubectl get --raw /k8s/clusters/<CLUSTER_ID>/apis/custom.metrics.k8s.io/v1beta1 # kubectl get --raw /k8s/clusters/<CLUSTER_ID>/apis/custom.metrics.k8s.io/v1beta1
{"kind":"APIResourceList","apiVersion":"v1","groupVersion":"custom.metrics.k8s.io/v1beta1","resources":[{"name":"pods/fs_usage_bytes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/memory_rss","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/spec_cpu_period","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/cpu_cfs_throttled","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_io_time","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_read","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_sector_writes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/cpu_user","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/last_seen","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/tasks_state","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/spec_cpu_quota","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/start_time_seconds","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_limit_bytes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_write","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/memory_cache","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/memory_usage_bytes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/cpu_cfs_periods","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/cpu_cfs_throttled_periods","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_reads_merged","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/memory_working_set_bytes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/network_udp_usage","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_inodes_free","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_inodes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_io_time_weighted","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/memory_failures","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/memory_swap","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/spec_cpu_shares","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/spec_memory_swap_limit_bytes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/cpu_usage","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_io_current","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_writes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/memory_failcnt","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_reads","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_writes_bytes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_writes_merged","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/network_tcp_usage","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/memory_max_usage_bytes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/spec_memory_limit_bytes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/spec_memory_reservation_limit_bytes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/cpu_load_average_10s","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/cpu_system","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_reads_bytes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_sector_reads","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]}]} {"kind":"APIResourceList","apiVersion":"v1","groupVersion":"custom.metrics.k8s.io/v1beta1","resources":[{"name":"pods/fs_usage_bytes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/memory_rss","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/spec_cpu_period","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/cpu_cfs_throttled","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_io_time","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_read","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_sector_writes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/cpu_user","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/last_seen","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/tasks_state","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/spec_cpu_quota","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/start_time_seconds","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_limit_bytes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_write","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/memory_cache","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/memory_usage_bytes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/cpu_cfs_periods","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/cpu_cfs_throttled_periods","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_reads_merged","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/memory_working_set_bytes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/network_udp_usage","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_inodes_free","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_inodes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_io_time_weighted","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/memory_failures","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/memory_swap","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/spec_cpu_shares","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/spec_memory_swap_limit_bytes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/cpu_usage","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_io_current","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_writes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/memory_failcnt","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_reads","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_writes_bytes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_writes_merged","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/network_tcp_usage","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/memory_max_usage_bytes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/spec_memory_limit_bytes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/spec_memory_reservation_limit_bytes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/cpu_load_average_10s","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/cpu_system","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_reads_bytes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_sector_reads","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]}]}
@@ -240,14 +272,15 @@ To do it, follow these steps:
#### ClusterRole and ClusterRoleBinding #### ClusterRole and ClusterRoleBinding
By default, hpa will try to read metrics (resource and custom) with user `system:anonymous`. It's needed to define `view-resource-metrics` and `view-custom-metrics` ClusterRole and ClusterRoleBindings assigning them to `system:anonymous` to open read access to metrics. By default, HPA reads resource and custom metrics with user `system:anonymous`. It's needed to define `view-resource-metrics` and `view-custom-metrics` ClusterRole and ClusterRoleBindings assigning them to `system:anonymous` to open read access to metrics.
To do it, follow these steps: To do it, follow these steps:
- Configure kubectl to connect proper k8s cluster. 1. Configure kubectl to connect proper k8s cluster.
- Copy ClusterRole and ClusterRoleBinding manifest for:
- resource metrics: ApiGroups `metrics.k8s.io` 1. Copy ClusterRole and ClusterRoleBinding manifest for.
``` {{% accordion id="resource-metrics" label="Resource Metrics: ApiGroups `metrics.k8s.io`" %}}
```
apiVersion: rbac.authorization.k8s.io/v1 apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole kind: ClusterRole
metadata: metadata:
@@ -275,9 +308,10 @@ To do it, follow these steps:
- apiGroup: rbac.authorization.k8s.io - apiGroup: rbac.authorization.k8s.io
kind: User kind: User
name: system:anonymous name: system:anonymous
``` ```
{{% /accordion %}}
- custom metrics: ApiGroups `custom.metrics.k8s.io` {{% accordion id="custom-resources" label="custom metrics: ApiGroups `custom.metrics.k8s.io`" %}}
``` ```
apiVersion: rbac.authorization.k8s.io/v1 apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole kind: ClusterRole
@@ -306,185 +340,184 @@ To do it, follow these steps:
kind: User kind: User
name: system:anonymous name: system:anonymous
``` ```
{{% /accordion %}}
- Create them at your k8s cluster (if want to use custom metrics) {{% accordion id="k8s-cluster" label="Create them at your k8s cluster (if want to use custom metrics)" %}}
``` ```
# kubectl create -f <RESOURCE_METRICS_MANIFEST> # kubectl create -f <RESOURCE_METRICS_MANIFEST>
# kubectl create -f <CUSTOM_METRICS_MANIFEST> # kubectl create -f <CUSTOM_METRICS_MANIFEST>
``` ```
{{% /accordion %}}
### Service deployment ### Service Deployment
To hpa works properly, service deployments should have resources request definition for containers. For HPA to work properly, service deployments should have resources request definitions for containers.
Lets see a hello-world example for testing if hpa is working fine. To do it, follow these steps:
- Configure kubectl to connect proper k8s cluster. Follow this hello-world example for testing if HPA is working.
- Copy `hello-world` deployment manifest.
``` 1. Configure kubectl to connect to your Kubernetes cluster.
apiVersion: apps/v1beta2
kind: Deployment 2. Copy the `hello-world` deployment manifest below.
metadata:
labels: apiVersion: apps/v1beta2
app: hello-world kind: Deployment
name: hello-world metadata:
namespace: default labels:
spec: app: hello-world
replicas: 1 name: hello-world
selector: namespace: default
matchLabels: spec:
app: hello-world replicas: 1
strategy: selector:
rollingUpdate: matchLabels:
maxSurge: 1 app: hello-world
maxUnavailable: 0 strategy:
type: RollingUpdate rollingUpdate:
template: maxSurge: 1
metadata: maxUnavailable: 0
labels: type: RollingUpdate
app: hello-world template:
spec: metadata:
containers: labels:
- image: rancher/hello-world app: hello-world
imagePullPolicy: Always spec:
containers:
- image: rancher/hello-world
imagePullPolicy: Always
name: hello-world
resources:
requests:
cpu: 500m
memory: 64Mi
ports:
- containerPort: 80
protocol: TCP
restartPolicy: Always
---
apiVersion: v1
kind: Service
metadata:
name: hello-world
namespace: default
spec:
ports:
- port: 80
protocol: TCP
targetPort: 80
selector:
app: hello-world
1. Deploy it at k8s cluster
```
# kubectl create -f <HELLO_WORLD_MANIFEST>
```
1. Copy hpa for resource or custom metrics:
{{% accordion id="resource metrics" label="resource metrics" %}}
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
name: hello-world
namespace: default
spec:
scaleTargetRef:
apiVersion: extensions/v1beta1
kind: Deployment
name: hello-world
minReplicas: 1
maxReplicas: 10
metrics:
- type: Resource
resource:
name: cpu
targetAverageUtilization: 50
- type: Resource
resource:
name: memory
targetAverageValue: 1000Mi
{{% /accordion %}}
{{% accordion id="custom-metrics" label="custom metrics (same as resource but adding custom cpu_system metric)" %}}
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
name: hello-world name: hello-world
resources: namespace: default
requests: spec:
cpu: 500m scaleTargetRef:
memory: 64Mi apiVersion: extensions/v1beta1
ports: kind: Deployment
- containerPort: 80 name: hello-world
protocol: TCP minReplicas: 1
restartPolicy: Always maxReplicas: 10
--- metrics:
apiVersion: v1 - type: Resource
kind: Service resource:
metadata: name: cpu
name: hello-world targetAverageUtilization: 50
namespace: default - type: Resource
spec: resource:
ports: name: memory
- port: 80 targetAverageValue: 100Mi
protocol: TCP - type: Pods
targetPort: 80 pods:
selector: metricName: cpu_system
app: hello-world targetAverageValue: 20m
``` {{% /accordion %}}
1. Getting hpa info and description and check that resource metrics data are shown.
{{% accordion id="resource-metrics" label="Resource Metrics" %}}
# kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
hello-world Deployment/hello-world 1253376 / 100Mi, 0% / 50% 1 10 1 6m
# kubectl describe hpa
Name: hello-world
Namespace: default
Labels: <none>
Annotations: <none>
CreationTimestamp: Mon, 23 Jul 2018 20:21:16 +0200
Reference: Deployment/hello-world
Metrics: ( current / target )
resource memory on pods: 1253376 / 100Mi
resource cpu on pods (as a percentage of request): 0% (0) / 50%
Min replicas: 1
Max replicas: 10
Conditions:
Type Status Reason Message
---- ------ ------ -------
AbleToScale True ReadyForNewScale the last scale time was sufficiently old as to warrant a new scale
ScalingActive True ValidMetricFound the HPA was able to successfully calculate a replica count from memory resource
ScalingLimited False DesiredWithinRange the desired count is within the acceptable range
Events: <none>
{{% /accordion %}}
{{% accordion id="custom metrics" label="Custom Metrics" %}}
# kubectl describe hpa
Name: hello-world
Namespace: default
Labels: <none>
Annotations: <none>
CreationTimestamp: Tue, 24 Jul 2018 18:36:28 +0200
Reference: Deployment/hello-world
Metrics: ( current / target )
resource memory on pods: 3514368 / 100Mi
"cpu_system" on pods: 0 / 20m
resource cpu on pods (as a percentage of request): 0% (0) / 50%
Min replicas: 1
Max replicas: 10
Conditions:
Type Status Reason Message
---- ------ ------ -------
AbleToScale True ReadyForNewScale the last scale time was sufficiently old as to warrant a new scale
ScalingActive True ValidMetricFound the HPA was able to successfully calculate a replica count from memory resource
ScalingLimited False DesiredWithinRange the desired count is within the acceptable range
Events: <none>
{{% /accordion %}}
- Deploy it at k8s cluster
```
# kubectl create -f <HELLO_WORLD_MANIFEST>
```
- Copy hpa for resource or custom metrics: 1. Generating load for the service to test up and down autoscalation. Any tool could be used at this point, but we've used `https://github.com/rakyll/hey` to generate http requests to our `hello-world` service, and observe if autoscaling is working propwrly.
- resource metrics 1. Observing autoscale up and down
```
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
name: hello-world
namespace: default
spec:
scaleTargetRef:
apiVersion: extensions/v1beta1
kind: Deployment
name: hello-world
minReplicas: 1
maxReplicas: 10
metrics:
- type: Resource
resource:
name: cpu
targetAverageUtilization: 50
- type: Resource
resource:
name: memory
targetAverageValue: 1000Mi
```
- custom metrics (same as resource but adding custom cpu_system metric)
```
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
name: hello-world
namespace: default
spec:
scaleTargetRef:
apiVersion: extensions/v1beta1
kind: Deployment
name: hello-world
minReplicas: 1
maxReplicas: 10
metrics:
- type: Resource
resource:
name: cpu
targetAverageUtilization: 50
- type: Resource
resource:
name: memory
targetAverageValue: 100Mi
- type: Pods
pods:
metricName: cpu_system
targetAverageValue: 20m
```
- Getting hpa info and description and check that resource metrics data are shown
- resource metrics
```
# kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
hello-world Deployment/hello-world 1253376 / 100Mi, 0% / 50% 1 10 1 6m
# kubectl describe hpa
Name: hello-world
Namespace: default
Labels: <none>
Annotations: <none>
CreationTimestamp: Mon, 23 Jul 2018 20:21:16 +0200
Reference: Deployment/hello-world
Metrics: ( current / target )
resource memory on pods: 1253376 / 100Mi
resource cpu on pods (as a percentage of request): 0% (0) / 50%
Min replicas: 1
Max replicas: 10
Conditions:
Type Status Reason Message
---- ------ ------ -------
AbleToScale True ReadyForNewScale the last scale time was sufficiently old as to warrant a new scale
ScalingActive True ValidMetricFound the HPA was able to successfully calculate a replica count from memory resource
ScalingLimited False DesiredWithinRange the desired count is within the acceptable range
Events: <none>
```
- custom metrics
```
# kubectl describe hpa
Name: hello-world
Namespace: default
Labels: <none>
Annotations: <none>
CreationTimestamp: Tue, 24 Jul 2018 18:36:28 +0200
Reference: Deployment/hello-world
Metrics: ( current / target )
resource memory on pods: 3514368 / 100Mi
"cpu_system" on pods: 0 / 20m
resource cpu on pods (as a percentage of request): 0% (0) / 50%
Min replicas: 1
Max replicas: 10
Conditions:
Type Status Reason Message
---- ------ ------ -------
AbleToScale True ReadyForNewScale the last scale time was sufficiently old as to warrant a new scale
ScalingActive True ValidMetricFound the HPA was able to successfully calculate a replica count from memory resource
ScalingLimited False DesiredWithinRange the desired count is within the acceptable range
Events: <none>
```
- Generating load for the service to test up and down autoscalation. Any tool could be used at this point, but we've used `https://github.com/rakyll/hey` to generate http requests to our `hello-world` service, and observe if autoscaling is working propwrly.
- Observing autoscale up and down
- Resource metrics - Resource metrics
Autoscale up to 2 pods when cpu usage is up to target Autoscale up to 2 pods when cpu usage is up to target