Merge pull request #2915 from superseb/rke_tolerations

Add information on configuring add-on tolerations
2026-05-17 10:25:16 +00:00 · 2020-12-15 11:10:59 -07:00
parent 3cf56e9add b55af3d4a7
commit 15c6b57c5f
5 changed files with 223 additions and 61 deletions
@@ -49,3 +49,9 @@ _Applies to v0.2.3 and higher_
 | kube-dns           | - `beta.kubernetes.io/os:NotIn:windows`<br/>- `node-role.kubernetes.io/worker` `Exists` | none  | - `NoSchedule:Exists`<br/>- `NoExecute:Exists`<br/>- `CriticalAddonsOnly:Exists` |
 | nginx-ingress      | - `beta.kubernetes.io/os:NotIn:windows`<br/>- `node-role.kubernetes.io/worker` `Exists` | none | - `NoSchedule:Exists`<br/>- `NoExecute:Exists` |
 | metrics-server     | - `beta.kubernetes.io/os:NotIn:windows`<br/>- `node-role.kubernetes.io/worker` `Exists` | none | - `NoSchedule:Exists`<br/>- `NoExecute:Exists` |
+
+## Tolerations
+
+_Available as of v1.2.4_
+
+Tolerations can be configured per add-on and apply to Deployment resources. The configured tolerations will replace the existing tolerations so make sure you configure all the tolerations you need. See the specific add-on doc pages for more information.
@@ -34,16 +34,16 @@ If you only want the CoreDNS pod to be deployed on specific nodes, you can set a

 ```yaml
 nodes:
-    - address: 1.1.1.1
-      role: [controlplane,worker,etcd]
-      user: root
-      labels:
-        app: dns
+- address: 1.1.1.1
+  role: [controlplane,worker,etcd]
+  user: root
+  labels:
+    app: dns

 dns:
-    provider: coredns
-    node_selector:
-      app: dns
+  provider: coredns
+  node_selector:
+    app: dns
 ```

 ## Configuring CoreDNS
@@ -56,10 +56,37 @@ When you set `upstreamnameservers`, the `provider` also needs to be set.

 ```yaml
 dns:
-    provider: coredns
-    upstreamnameservers:
-    - 1.1.1.1
-    - 8.8.4.4
+  provider: coredns
+  upstreamnameservers:
+  - 1.1.1.1
+  - 8.8.4.4
+```
+
+### Tolerations
+
+_Available as of v1.2.4_
+
+The configured tolerations apply to the `coredns` and the `coredns-autoscaler` Deployment.
+
+```
+dns:
+  provider: coredns
+  tolerations:
+  - key: "node.kubernetes.io/unreachable"
+    operator: "Exists"
+    effect: "NoExecute"
+    tolerationseconds: 300
+  - key: "node.kubernetes.io/not-ready"
+    operator: "Exists"
+    effect: "NoExecute"
+    tolerationseconds: 300
+```
+
+To check for applied tolerations on the `coredns` and `coredns-autoscaler` Deployment, use the following commands:
+
+```
+kubectl -n kube-system get deploy coredns -o jsonpath='{.spec.template.spec.tolerations}'
+kubectl -n kube-system get deploy coredns-autoscaler -o jsonpath='{.spec.template.spec.tolerations}'
 ```

 # kube-dns
@@ -76,16 +103,16 @@ If you only want the kube-dns pod to be deployed on specific nodes, you can set

 ```yaml
 nodes:
-    - address: 1.1.1.1
-      role: [controlplane,worker,etcd]
-      user: root
-      labels:
-        app: dns
+- address: 1.1.1.1
+  role: [controlplane,worker,etcd]
+  user: root
+  labels:
+    app: dns

 dns:
-    provider: kube-dns
-    node_selector:
-      app: dns
+  provider: kube-dns
+  node_selector:
+    app: dns
 ```

 ## Configuring kube-dns
@@ -100,10 +127,38 @@ When you set `upstreamnameservers`, the `provider` also needs to be set.

 ```yaml
 dns:
-    provider: kube-dns
-    upstreamnameservers:
-    - 1.1.1.1  
-    - 8.8.4.4
+  provider: kube-dns
+  upstreamnameservers:
+  - 1.1.1.1
+  - 8.8.4.4
+```
+
+### Tolerations
+
+_Available as of v1.2.4_
+
+The configured tolerations apply to the `kube-dns` and the `kube-dns-autoscaler` Deployment.
+
+```
+dns:
+  provider: kube-dns
+  tolerations:
+  - key: "node.kubernetes.io/unreachable"
+    operator: "Exists"
+    effect: "NoExecute"
+    tolerationseconds: 300
+  - key: "node.kubernetes.io/not-ready"
+    operator: "Exists"
+    effect: "NoExecute"
+    tolerationseconds: 300
+
+```
+
+To check for applied tolerations on the `coredns` and `coredns-autoscaler` Deployment, use the following commands:
+
+```
+kubectl get deploy kube-dns -n kube-system -o jsonpath='{.spec.template.spec.tolerations}'
+kubectl get deploy kube-dns-autoscaler -n kube-system -o jsonpath='{.spec.template.spec.tolerations}'
 ```

 # Disabling deployment of a DNS provider
@@ -114,7 +169,7 @@ You can disable the default DNS provider by specifying `none` to  the dns `provi

 ```yaml
 dns:
-    provider: none
+  provider: none
 ```

 # NodeLocal DNS
@@ -137,9 +192,9 @@ The `ip_address` parameter is used to configure what link-local IP address will

 ```yaml
 dns:
-    provider: coredns
-    nodelocal:
-        ip_address: "169.254.20.10"
+  provider: coredns
+  nodelocal:
+    ip_address: "169.254.20.10"
 ```

 > **Note:** When enabling NodeLocal DNS on an existing cluster, pods that are currently running will not be modified, the updated `/etc/resolv.conf` configuration will take effect only for pods started after enabling NodeLocal DNS.
@@ -18,16 +18,41 @@ If you only wanted ingress controllers to be deployed on specific nodes, you can

 ```yaml
 nodes:
-    - address: 1.1.1.1
-      role: [controlplane,worker,etcd]
-      user: root
-      labels:
-        app: ingress
+- address: 1.1.1.1
+  role: [controlplane,worker,etcd]
+  user: root
+  labels:
+    app: ingress

 ingress:
-    provider: nginx
-    node_selector:
-      app: ingress
+  provider: nginx
+  node_selector:
+    app: ingress
+```
+
+## Tolerations
+
+_Available as of v1.2.4_
+
+The configured tolerations apply to the `default-http-backend` Deployment.
+
+```
+ingress:
+  tolerations:
+  - key: "node.kubernetes.io/unreachable"
+    operator: "Exists"
+    effect: "NoExecute"
+    tolerationseconds: 300
+  - key: "node.kubernetes.io/not-ready"
+    operator: "Exists"
+    effect: "NoExecute"
+    tolerationseconds: 300
+```
+
+To check for applied tolerations `default-http-backend` Deployment, use the following commands:
+
+```
+kubectl -n ingress-nginx get deploy default-http-backend -o jsonpath='{.spec.template.spec.tolerations}'
 ```

 ## Disabling the Default Ingress Controller
@@ -44,12 +69,12 @@ For the configuration of NGINX, there are configuration options available in Kub

 ```yaml
 ingress:
-    provider: nginx
-    options:
-      map-hash-bucket-size: "128"
-      ssl-protocols: SSLv2
-    extra_args:
-      enable-ssl-passthrough: ""
+  provider: nginx
+  options:
+    map-hash-bucket-size: "128"
+    ssl-protocols: SSLv2
+  extra_args:
+    enable-ssl-passthrough: ""
 ```

 ### Disabling NGINX Ingress Default Backend
@@ -58,7 +83,7 @@ As of v0.20.0, you can disable the [default backend service](https://kubernetes.

 ```yaml
 ingress:
-    default_backend: false
+  default_backend: false
 ```

 > **What happens if the field is omitted?** The value of `default_backend` will default to `true`. This maintains behavior with older versions of `rke`. However, a future version of `rke` will change the default value to `false`.
@@ -9,6 +9,31 @@ RKE will deploy Metrics Server as a Deployment.

 The image used for Metrics Server is under the [`system_images` directive]({{<baseurl>}}/rke/latest/en/config-options/system-images/). For each Kubernetes version, there is a default image associated with the Metrics Server, but these can be overridden by changing the image tag in `system_images`.

+## Tolerations
+
+_Available as of v1.2.4_
+
+The configured tolerations apply to the `metrics-server` Deployment.
+
+```
+monitoring:
+  tolerations:
+  - key: "node.kubernetes.io/unreachable"
+    operator: "Exists"
+    effect: "NoExecute"
+    tolerationseconds: 300
+  - key: "node.kubernetes.io/not-ready"
+    operator: "Exists"
+    effect: "NoExecute"
+    tolerationseconds: 300
+```
+
+To check for applied tolerations on the `metrics-server` Deployment, use the following commands:
+
+```
+kubectl -n kube-system get deploy metrics-server -o jsonpath='{.spec.template.spec.tolerations}'
+```
+
 ## Disabling the Metrics Server

 _Available as of v0.2.0_
@@ -17,5 +42,5 @@ You can disable the default controller by specifying `none` to the monitoring `p

 ```yaml
 monitoring:
-    provider: none
+  provider: none
 ```
@@ -17,7 +17,7 @@ By default, the network plug-in is `canal`. If you want to use another network p
 ```yaml
 # Setting the flannel network plug-in
 network:
-    plugin: flannel
+  plugin: flannel
 ```

 The images used for network plug-ins are under the [`system_images` directive]({{<baseurl>}}/rke/latest/en/config-options/system-images/). For each Kubernetes version, there are default images associated with each network plug-in, but these can be overridden by changing the image tag in `system_images`.
@@ -28,7 +28,7 @@ You can disable deploying a network plug-in by specifying `none` to the network

 ```yaml
 network:
-    plugin: none
+  plugin: none
 ```

 # Network Plug-in Options
@@ -39,10 +39,10 @@ Besides the different images that could be used to deploy network plug-ins, cert

 ```yaml
 network:
-    plugin: canal
-    options:
-        canal_iface: eth1
-        canal_flannel_backend_type: vxlan
+  plugin: canal
+  options:
+    canal_iface: eth1
+    canal_flannel_backend_type: vxlan
 ```

 #### Canal Interface
@@ -50,14 +50,40 @@ network:
 By setting the `canal_iface`, you can configure the interface to use for inter-host communication.
 The `canal_flannel_backend_type` option allows you to specify the type of [flannel backend](https://github.com/coreos/flannel/blob/master/Documentation/backends.md) to use. By default the `vxlan` backend is used.

+## Canal Network Plug-in Tolerations
+
+_Available as of v1.2.4_
+
+The configured tolerations apply to the `calico-kube-controllers` Deployment.
+
+```
+network:
+  plugin: canal
+  tolerations:
+  - key: "node.kubernetes.io/unreachable"
+    operator: "Exists"
+    effect: "NoExecute"
+    tolerationseconds: 300
+  - key: "node.kubernetes.io/not-ready"
+    operator: "Exists"
+    effect: "NoExecute"
+    tolerationseconds: 300
+```
+
+To check for applied tolerations on the `calico-kube-controllers` Deployment, use the following command:
+
+```
+kubectl -n kube-system get deploy calico-kube-controllers -o jsonpath='{.spec.template.spec.tolerations}'
+```
+
 ## Flannel Network Plug-in Options

 ```yaml
 network:
-    plugin: flannel
-    options:
-        flannel_iface: eth1
-        flannel_backend_type: vxlan
+  plugin: flannel
+  options:
+    flannel_iface: eth1
+    flannel_backend_type: vxlan
 ```

 #### Flannel Interface
@@ -69,9 +95,9 @@ The `flannel_backend_type` option allows you to specify the type of [flannel bac

 ```yaml
 network:
-    plugin: calico
-    options:
-        calico_cloud_provider: aws
+  plugin: calico
+  options:
+    calico_cloud_provider: aws
 ```
 #### Calico Cloud Provider

@@ -82,20 +108,45 @@ Calico currently only supports 2 cloud providers, AWS or GCE, which can be set u
 - `aws`
 - `gce`

+## Calico Network Plug-in Tolerations
+
+_Available as of v1.2.4_
+
+The configured tolerations apply to the `calico-kube-controllers` Deployment.
+
+```
+network:
+  plugin: calico
+  tolerations:
+  - key: "node.kubernetes.io/unreachable"
+    operator: "Exists"
+    effect: "NoExecute"
+    tolerationseconds: 300
+  - key: "node.kubernetes.io/not-ready"
+    operator: "Exists"
+    effect: "NoExecute"
+    tolerationseconds: 300
+```
+
+To check for applied tolerations on the `calico-kube-controllers` Deployment, use the following command:
+
+```
+kubectl -n kube-system get deploy calico-kube-controllers -o jsonpath='{.spec.template.spec.tolerations}'
+```
+
 ## Weave Network Plug-in Options

 ```yaml
 network:
-    plugin: weave
-    weave_network_provider:
-        password: "Q]SZOQ5wp@n$oijz"
+  plugin: weave
+  weave_network_provider:
+    password: "Q]SZOQ5wp@n$oijz"
 ```

 #### Weave encryption

 Weave encryption can be enabled by passing a string password to the network provider config.

-
 ## Custom Network Plug-ins

 It is possible to add a custom network plug-in by using the [user-defined add-on functionality]({{<baseurl>}}/rke/latest/en/config-options/add-ons/user-defined-add-ons/) of RKE. In the `addons` field, you can add the add-on manifest of a cluster that has the network plugin-that you want, as shown in [this example.]({{<baseurl>}}/rke/latest/en/config-options/add-ons/network-plugins/custom-network-plugin-example)