public/rancher-docs
1
0
Fork 0
mirror of https://github.com/rancher/rancher-docs.git synced 2026-05-04 20:23:24 +00:00
Code Issues Packages Projects Releases Wiki Activity

Add the instruction for enable API audit log in downstream clusters (#824)

Browse Source
This commit is contained in:
Jiaqi Luo
2023-09-13 16:05:50 -07:00
committed by GitHub
parent f6a70125b1
commit bcfd5b25aa
8 changed files with 1250 additions and 206 deletions
Download Patch File Download Diff File Expand all files Collapse all files
versioned_docs/version-2.7/reference-guides/cluster-configuration/rancher-server-configuration/k3s-cluster-configuration.md
+316 -37
View File
@@ -15,94 +15,158 @@ You can configure the Kubernetes options one of two ways:
- [Rancher UI](#configuration-options-in-the-rancher-ui): Use the Rancher UI to select options that are commonly customized when setting up a Kubernetes cluster.
- [Cluster Config File](#cluster-config-file): Instead of using the Rancher UI to choose Kubernetes options for the cluster, advanced users can create a K3s config file. Using a config file allows you to set any of the [options](https://rancher.com/docs/k3s/latest/en/installation/install-options/) available in an K3s installation.
## Editing Clusters in the Rancher UI
The Rancher UI provides two ways to edit a cluster:
1. With a form.
1. With YAML.
### Editing Clusters with a Form
The form covers the most frequently needed options for clusters.
To edit your cluster,
1. Click **☰ > Cluster Management**.
1. Go to the cluster you want to configure and click **⋮ > Edit Config**.
### Editing Clusters in YAML
For a complete reference of configurable options for K3s clusters in YAML, see the [K3s documentation.](https://rancher.com/docs/k3s/latest/en/installation/install-options/)
To edit your cluster with YAML:
1. Click **☰ > Cluster Management**.
1. Go to the cluster you want to configure and click **⋮ > Edit as YAML**.
1. Edit the RKE options under the `rkeConfig` directive.
## Configuration Options in the Rancher UI
:::tip
### Machine Pool Configuration
Some advanced configuration options are not exposed in the Rancher UI forms, but they can be enabled by editing the K3s cluster configuration file in YAML. For the complete reference of configurable options for K3s clusters in YAML, see the [K3s documentation.](https://rancher.com/docs/k3s/latest/en/installation/install-options/)
This subsection covers generic machine pool configurations. For specific infrastructure provider configurations, refer to the following:
:::
- [Azure](../downstream-cluster-configuration/machine-configuration/azure.md)
- [DigitalOcean](../downstream-cluster-configuration/machine-configuration/digitalocean.md)
- [EC2](../downstream-cluster-configuration/machine-configuration/amazon-ec2.md)
### Basics
#### Kubernetes Version
##### Pool Name
The name of the machine pool.
##### Machine Count
The number of machines in the pool.
##### Roles
Option to assign etcd, control plane, and worker roles to nodes.
#### Advanced
##### Auto Replace
The amount of time nodes can be unreachable before they are automatically deleted and replaced.
##### Drain Before Delete
Enables draining nodes by evicting all pods before the node is deleted.
##### Kubernetes Node Labels
Add [labels](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/) to nodes to help with organization and object selection.
For details on label syntax requirements, see the [Kubernetes documentation.](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#syntax-and-character-set)
##### Taints
Add [taints](https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/) to nodes, to prevent pods from being scheduled to or executed on the nodes, unless the pods have matching tolerations.
### Cluster Configuration
#### Basics
##### Kubernetes Version
The version of Kubernetes installed on your cluster nodes. Rancher packages its own version of Kubernetes based on [hyperkube](https://github.com/rancher/hyperkube).
For more detail, see [Upgrading Kubernetes](../../../getting-started/installation-and-upgrade/upgrade-and-roll-back-kubernetes.md).
#### Encrypt Secrets
##### Pod Security Admission Configuration Template
The default [pod security admission configuration template](../../../how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/psa-config-templates.md) for the cluster.
##### Encrypt Secrets
Option to enable or disable secrets encryption. When enabled, secrets will be encrypted using a AES-CBC key. If disabled, any previously secrets will not be readable until encryption is enabled again. Refer to the [K3s documentation](https://rancher.com/docs/k3s/latest/en/advanced/#secrets-encryption-config-experimental) for details.
#### Project Network Isolation
##### Project Network Isolation
If your network provider allows project network isolation, you can choose whether to enable or disable inter-project communication.
#### SELinux
##### SELinux
Option to enable or disable [SELinux](https://rancher.com/docs/k3s/latest/en/advanced/#selinux-support) support.
#### CoreDNS
##### CoreDNS
By default, [CoreDNS](https://coredns.io/) is installed as the default DNS provider. If CoreDNS is not installed, an alternate DNS provider must be installed yourself. Refer to the [K3s documentation](https://rancher.com/docs/k3s/latest/en/networking/#coredns) for details..
#### Klipper Service LB
##### Klipper Service LB
Option to enable or disable the [Klipper](https://github.com/rancher/klipper-lb) service load balancer. Refer to the [K3s documentation](https://rancher.com/docs/k3s/latest/en/networking/#service-load-balancer) for details.
#### Traefik Ingress
##### Traefik Ingress
Option to enable or disable the [Traefik](https://traefik.io/) HTTP reverse proxy and load balancer. For more details and configuration options, see the [K3s documentation](https://rancher.com/docs/k3s/latest/en/networking/#traefik-ingress-controller).
#### Local Storage
##### Local Storage
Option to enable or disable [local storage](https://rancher.com/docs/k3s/latest/en/storage/) on the node(s).
#### Metrics Server
##### Metrics Server
Option to enable or disable the [metrics server](https://github.com/kubernetes-incubator/metrics-server). If enabled, ensure port 10250 is opened for inbound TCP traffic.
### Add-On Config
#### Add-On Config
Additional Kubernetes manifests, managed as a [Add-on](https://kubernetes.io/docs/concepts/cluster-administration/addons/), to apply to the cluster on startup. Refer to the [K3s documentation](https://rancher.com/docs/k3s/latest/en/helm/#automatically-deploying-manifests-and-helm-charts) for details.
### Agent Environment Vars
#### Agent Environment Vars
Option to set environment variables for [K3s agents](https://rancher.com/docs/k3s/latest/en/architecture/). The environment variables can be set using key value pairs. Refer to the [K3 documentation](https://rancher.com/docs/k3s/latest/en/installation/install-options/agent-config/) for more details.
### etcd
#### etcd
#### Automatic Snapshots
##### Automatic Snapshots
Option to enable or disable recurring etcd snapshots. If enabled, users have the option to configure the frequency of snapshots. For details, refer to the [K3s documentation](https://rancher.com/docs/k3s/latest/en/backup-restore/#creating-snapshots).
#### Metrics
##### Metrics
Option to choose whether to expose etcd metrics to the public or only within the cluster.
### Networking
#### Networking
#### Cluster CIDR
##### Cluster CIDR
IPv4/IPv6 network CIDRs to use for pod IPs (default: 10.42.0.0/16).
#### Service CIDR
##### Service CIDR
IPv4/IPv6 network CIDRs to use for service IPs (default: 10.43.0.0/16).
#### Cluster DNS
##### Cluster DNS
IPv4 Cluster IP for coredns service. Should be in your service-cidr range (default: 10.43.0.10).
#### Cluster Domain
##### Cluster Domain
Select the domain for the cluster. The default is `cluster.local`.
#### NodePort Service Port Range
##### NodePort Service Port Range
Option to change the range of ports that can be used for [NodePort services](https://kubernetes.io/docs/concepts/services-networking/service/#nodeport). The default is `30000-32767`.
#### Truncate Hostnames
##### Truncate Hostnames
Option to truncate hostnames to 15 characters or less. You can only set this field during the initial creation of the cluster. You can't enable or disable the 15 character limit after cluster creation.
@@ -110,11 +174,11 @@ This setting only affects machine-provisioned clusters. Since custom clusters se
Truncating hostnames in a cluster improves compatibility with Windows-based systems. Although Kubernetes allows hostnames up to 63 characters in length, systems that use NetBIOS restrict hostnames to 15 characters or less.
#### TLS Alternate Names
##### TLS Alternate Names
Add additional hostnames or IPv4/IPv6 addresses as Subject Alternative Names on the server TLS cert.
#### Authorized Cluster Endpoint
##### Authorized Cluster Endpoint
Authorized Cluster Endpoint can be used to directly access the Kubernetes API server, without requiring communication through Rancher.
@@ -122,34 +186,249 @@ For more detail on how an authorized cluster endpoint works and why it is used,
We recommend using a load balancer with the authorized cluster endpoint. For details, refer to the [recommended architecture section.](../../rancher-manager-architecture/architecture-recommendations.md#architecture-for-an-authorized-cluster-endpoint-ace)
### Registries
#### Registries
Select the image repository to pull Rancher images from. For more details and configuration options, see the [K3s documentation](https://rancher.com/docs/k3s/latest/en/installation/private-registry/).
### Upgrade Strategy
#### Upgrade Strategy
#### Controle Plane Concurrency
##### Control Plane Concurrency
Select how many nodes can be upgraded at the same time. Can be a fixed number or percentage.
#### Worker Concurrency
##### Worker Concurrency
Select how many nodes can be upgraded at the same time. Can be a fixed number or percentage.
#### Drain Nodes (Control Plane)
##### Drain Nodes (Control Plane)
Option to remove all pods from the node prior to upgrading.
#### Drain Nodes (Worker Nodes)
##### Drain Nodes (Worker Nodes)
Option to remove all pods from the node prior to upgrading.
### Advanced
#### Advanced
Option to set kubelet options for different nodes. For available options, refer to the [Kubernetes documentation](https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/).
## Cluster Config File
## Cluster Config File Reference
Instead of using the Rancher UI forms to choose Kubernetes options for the cluster, advanced users can create an K3s config file. Using a config file allows you to set any of the [options](https://rancher.com/docs/k3s/latest/en/installation/install-options/) available in an K3s installation.
Editing clusters in YAML allows you to set configurations that are already listed in [Configuration Options in the Rancher UI](#configuration-options-in-the-rancher-ui), as well as set Rancher-specific parameters.
To edit an K3s config file directly from the Rancher UI, click **Edit as YAML**.
<details>
<summary>
<b>Example Cluster Config File Snippet</b>
</summary>
```yaml
apiVersion: provisioning.cattle.io/v1
kind: Cluster
spec:
cloudCredentialSecretName: cattle-global-data:cc-fllv6
clusterAgentDeploymentCustomization: {}
fleetAgentDeploymentCustomization: {}
kubernetesVersion: v1.26.7+k3s1
localClusterAuthEndpoint: {}
rkeConfig:
chartValues: {}
etcd:
snapshotRetention: 5
snapshotScheduleCron: 0 */5 * * *
machineGlobalConfig:
disable-apiserver: false
disable-cloud-controller: false
disable-controller-manager: false
disable-etcd: false
disable-kube-proxy: false
disable-network-policy: false
disable-scheduler: false
etcd-expose-metrics: false
kube-apiserver-arg:
- audit-policy-file=/etc/rancher/k3s/user-audit-policy.yaml
- audit-log-path=/etc/rancher/k3s/user-audit.logs
profile: null
secrets-encryption: false
machinePools:
- controlPlaneRole: true
etcdRole: true
machineConfigRef:
kind: Amazonec2Config
name: nc-test-pool1-pwl5h
name: pool1
quantity: 1
unhealthyNodeTimeout: 0s
workerRole: true
machineSelectorConfig:
- config:
docker: false
protect-kernel-defaults: false
selinux: false
machineSelectorFiles:
- fileSources:
- configMap:
name: ''
secret:
name: audit-policy
items:
- key: audit-policy
path: /etc/rancher/k3s/user-audit-policy.yaml
machineLabelSelector:
matchLabels:
rke.cattle.io/control-plane-role: 'true'
registries: {}
upgradeStrategy:
controlPlaneConcurrency: '1'
controlPlaneDrainOptions:
deleteEmptyDirData: true
disableEviction: false
enabled: false
force: false
gracePeriod: -1
ignoreDaemonSets: true
ignoreErrors: false
postDrainHooks: null
preDrainHooks: null
skipWaitForDeleteTimeoutSeconds: 0
timeout: 120
workerConcurrency: '1'
workerDrainOptions:
deleteEmptyDirData: true
disableEviction: false
enabled: false
force: false
gracePeriod: -1
ignoreDaemonSets: true
ignoreErrors: false
postDrainHooks: null
preDrainHooks: null
skipWaitForDeleteTimeoutSeconds: 0
timeout: 120
```
</details>
### chartValues
Specify the values for the system charts installed by K3s.
Example:
```yaml
chartValues:
chart-name:
key: value
```
### machineGlobalConfig
Specify K3s configurations. Any configuration change made here will apply to every node. The configuration options available in the [standalone version of k3s](https://docs.k3s.io/cli/server) can be applied here.
Example:
```yaml
machineGlobalConfig:
etcd-arg:
- key1=value1
- key2=value2
```
### machineSelectorConfig
`machineSelectorConfig` is the same as [`machineGlobalConfig`](#machineglobalconfig) except that a [label](#kubernetes-node-labels) selector can be specified with the configuration. The configuration will only be applied to nodes that match the provided label selector.
Multiple `config` entries are allowed, each specifying their own `machineLabelSelector`. A user can specify `matchExpressions`, `matchLabels`, both, or neither. Omitting the `machineLabelSelector` section of this field has the same effect as putting the config in the `machineGlobalConfig` section.
Example:
```yaml
machineSelectorConfig
- config:
config-key: config-value
machineLabelSelector:
matchExpressions:
- key: example-key
operator: string # Valid operators are In, NotIn, Exists and DoesNotExist.
values:
- example-value1
- example-value2
matchLabels:
key1: value1
key2: value2
```
### machineSelectorFiles
:::note
This feature is available in Rancher v2.7.2 and later.
:::
Deliver files to nodes, so that the files can be in place before initiating K3s server or agent processes.
The content of the file is retrieved from either a secret or a configmap. The target nodes are filtered by the `machineLabelSelector`.
Example :
```yaml
machineSelectorFiles:
- fileSources:
- secret:
items:
- key: example-key
path: path-to-put-the-file-on-nodes
permissions: 644 (optional)
hash: base64-encoded-hash-of-the-content (optional)
name: example-secret-name
machineLabelSelector:
matchExpressions:
- key: example-key
operator: string # Valid operators are In, NotIn, Exists and DoesNotExist.
values:
- example-value1
- example-value2
matchLabels:
key1: value1
key2: value2
- fileSources:
- configMap:
items:
- key: example-key
path: path-to-put-the-file-on-nodes
permissions: 644 (optional)
hash: base64-encoded-hash-of-the-content (optional)
name: example-configmap-name
machineLabelSelector:
matchExpressions:
- key: example-key
operator: string # Valid operators are In, NotIn, Exists and DoesNotExist.
values:
- example-value1
- example-value2
matchLabels:
key1: value1
key2: value2
```
The secret or configmap must meet the following requirements:
1. It must be in the `fleet-default` namespace where the Cluster object exists.
2. It must have the annotation `rke.cattle.io/object-authorized-for-clusters: cluster-name1,cluster-name2`, which permits the target clusters to use it.
:::tip
Rancher Dashboard provides an easy-to-use form for creating the secret or configmap.
:::
Example:
```yaml
apiVersion: v1
data:
audit-policy: >-
IyBMb2cgYWxsIHJlcXVlc3RzIGF0IHRoZSBNZXRhZGF0YSBsZXZlbC4KYXBpVmVyc2lvbjogYXVkaXQuazhzLmlvL3YxCmtpbmQ6IFBvbGljeQpydWxlczoKLSBsZXZlbDogTWV0YWRhdGE=
kind: Secret
metadata:
annotations:
rke.cattle.io/object-authorized-for-clusters: cluster1
name: name1
namespace: fleet-default
```
versioned_docs/version-2.7/reference-guides/cluster-configuration/rancher-server-configuration/rke2-cluster-configuration.md
+166 -66
View File
@@ -15,18 +15,26 @@ You can configure the Kubernetes options in one of the two following ways:
- [Rancher UI](#configuration-options-in-the-rancher-ui): Use the Rancher UI to select options that are commonly customized when setting up a Kubernetes cluster.
- [Cluster Config File](#cluster-config-file-reference): Instead of using the Rancher UI to choose Kubernetes options for the cluster, advanced users can create an RKE2 config file. Using a config file allows you to set many additional [options](https://docs.rke2.io/install/configuration) available for an RKE2 installation.
## Editing Clusters with a Form in the Rancher UI
## Editing Clusters in the Rancher UI
The Rancher UI provides two ways to edit a cluster:
1. With a form.
1. With YAML.
### Editing Clusters with a Form
The form covers the most frequently needed options for clusters.
To edit your cluster,
1. In the upper left corner, click **☰ > Cluster Management**.
1. Click **☰ > Cluster Management**.
1. Go to the cluster you want to configure and click **⋮ > Edit Config**.
## Editing Clusters with YAML
### Editing Clusters in YAML
Instead of using the Rancher UI to choose Kubernetes options for the cluster, advanced users can create an RKE2 config file. Using a config file allows you to set any of the options available in an RKE2 installation by specifying them in YAML.
For a complete reference of configurable options for K3s clusters in YAML, see the [K3s documentation.](https://rancher.com/docs/k3s/latest/en/installation/install-options/)
To edit an RKE2 config file directly from the Rancher UI,
To edit your cluster in YAML:
1. Click **☰ > Cluster Management**.
1. Go to the cluster you want to configure and click **⋮ > Edit as YAML**.
@@ -34,62 +42,55 @@ To edit an RKE2 config file directly from the Rancher UI,
## Configuration Options in the Rancher UI
:::tip
### Machine Pool Configuration
Some advanced configuration options are not exposed in the Rancher UI forms, but they can be enabled by editing the RKE2 cluster configuration file in YAML. For the complete reference of configurable options for RKE2 Kubernetes clusters in YAML, see the [RKE2 documentation.](https://docs.rke2.io/install/configuration)
:::
## Machine Pool
This subsection covers the generic machine pool configurations. For infrastructure provider specific, configurations refer to the following pages:
This subsection covers generic machine pool configurations. For specific infrastructure provider configurations, refer to the following:
- [Azure](../downstream-cluster-configuration/machine-configuration/azure.md)
- [DigitalOcean](../downstream-cluster-configuration/machine-configuration/digitalocean.md)
- [EC2](../downstream-cluster-configuration/machine-configuration/amazon-ec2.md)
### Pool Name
##### Pool Name
The name of the machine pool.
### Machine Count
##### Machine Count
The number of machines in the pool.
### Roles
##### Roles
Option to assign etcd, control plane, and worker roles to nodes.
### Advanced
#### Advanced
#### Auto Replace
##### Auto Replace
The duration nodes can be unreachable before they are automatically deleted and replaced.
The amount of time nodes can be unreachable before they are automatically deleted and replaced.
#### Drain Before Delete
##### Drain Before Delete
Enables draining nodes by evicting all pods before the node is deleted.
#### Kubernetes Node Labels
##### Kubernetes Node Labels
Add [labels](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/) to nodes to help with organization and object selection.
For details on label syntax requirements, see the [Kubernetes documentation.](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#syntax-and-character-set)
#### Taints
##### Taints
Add [taints](https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/) to nodes, which can be used to prevent pods from being scheduled to or executed on nodes, unless the pods have matching tolerations.
Add [taints](https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/) to nodes, to prevent pods from being scheduled to or executed on the nodes, unless the pods have matching tolerations.
## Cluster Configuration
### Basics
#### Kubernetes Version
### Cluster Configuration
#### Basics
##### Kubernetes Version
The version of Kubernetes installed on your cluster nodes. Rancher packages its own version of Kubernetes based on [hyperkube](https://github.com/rancher/hyperkube).
For more detail, see [Upgrading Kubernetes](../../../getting-started/installation-and-upgrade/upgrade-and-roll-back-kubernetes.md).
#### Container Network Provider
##### Container Network Provider
The [Network Provider](https://kubernetes.io/docs/concepts/cluster-administration/networking/) that the cluster uses.
@@ -110,7 +111,7 @@ Out of the box, Rancher is compatible with the following network providers:
For more details on the different networking providers and how to configure them, please view our [RKE2 documentation](https://docs.rke2.io/install/network_options).
##### Dual-stack Networking
###### Dual-stack Networking
[Dual-stack](https://docs.rke2.io/install/network_options#dual-stack-configuration) networking is supported for all CNI providers. To configure RKE2 in dual-stack mode, set valid IPv4/IPv6 CIDRs for your [Cluster CIDR](#cluster-cidr) and/or [Service CIDR](#service-cidr).
@@ -118,7 +119,7 @@ For more details on the different networking providers and how to configure them
When using `cilium` or `multus,cilium` as your container network interface provider, ensure the **Enable IPv6 Support** option is also enabled.
#### Cloud Provider
##### Cloud Provider
You can configure a [Kubernetes cloud provider](../../../pages-for-subheaders/set-up-cloud-providers.md). If you want to use dynamically provisioned [volumes and storage](../../../pages-for-subheaders/create-kubernetes-persistent-storage.md) in Kubernetes, typically you must select the specific cloud provider in order to use it. For example, if you want to use Amazon EBS, you would need to select the `aws` cloud provider.
@@ -128,89 +129,93 @@ If the cloud provider you want to use is not listed as an option, you will need
:::
#### Default Pod Security Policy
##### Default Pod Security Policy
Choose the default [pod security policy](../../../how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/create-pod-security-policies.md) for the cluster. Please refer to the [RKE2 documentation](https://docs.rke2.io/security/pod_security_policies) on the specifications of each available policy.
The default [pod security policy](../../../how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/create-pod-security-policies.md) for the cluster. Please refer to the [RKE2 documentation](https://docs.rke2.io/security/pod_security_policies) on the specifications of each available policy.
#### Worker CIS Profile
##### Pod Security Admission Configuration Template
The default [pod security admission configuration template](../../../how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/psa-config-templates.md) for the cluster.
##### Worker CIS Profile
Select a [CIS benchmark](../../../pages-for-subheaders/cis-scan-guides.md) to validate the system configuration against.
#### Project Network Isolation
##### Project Network Isolation
If your network provider allows project network isolation, you can choose whether to enable or disable inter-project communication.
Project network isolation is available if you are using any RKE2 network plugin that supports the enforcement of Kubernetes network policies, such as Canal.
#### CoreDNS
##### CoreDNS
By default, [CoreDNS](https://coredns.io/) is installed as the default DNS provider. If CoreDNS is not installed, an alternate DNS provider must be installed yourself. Refer to the [RKE2 documentation](https://docs.rke2.io/networking#coredns) for additional CoreDNS configurations.
#### NGINX Ingress
##### NGINX Ingress
If you want to publish your applications in a high-availability configuration, and you're hosting your nodes with a cloud-provider that doesn't have a native load-balancing feature, enable this option to use NGINX Ingress within the cluster. Refer to the [RKE2 documentation](https://docs.rke2.io/networking#nginx-ingress-controller) for additional configuration options.
Refer to the [RKE2 documentation](https://docs.rke2.io/networking#nginx-ingress-controller) for additional configuration options.
#### Metrics Server
##### Metrics Server
Option to enable or disable [Metrics Server](https://rancher.com/docs/rke/latest/en/config-options/add-ons/metrics-server/).
Each cloud provider capable of launching a cluster using RKE2 can collect metrics and monitor for your cluster nodes. Enable this option to view your node metrics from your cloud provider's portal.
### Add-On Config
#### Add-On Config
Additional Kubernetes manifests, managed as an [Add-on](https://kubernetes.io/docs/concepts/cluster-administration/addons/), to apply to the cluster on startup. Refer to the [RKE2 documentation](https://docs.rke2.io/helm#automatically-deploying-manifests-and-helm-charts) for details.
### Agent Environment Vars
#### Agent Environment Vars
Option to set environment variables for [Rancher agents](../../../how-to-guides/new-user-guides/launch-kubernetes-with-rancher/about-rancher-agents.md). The environment variables can be set using key value pairs. Refer to the [RKE2 documentation](https://docs.rke2.io/reference/linux_agent_config) for more details.
### etcd
#### etcd
#### Automatic Snapshots
##### Automatic Snapshots
Option to enable or disable recurring etcd snapshots. If enabled, users have the option to configure the frequency of snapshots. For details, refer to the [RKE2 documentation](https://docs.rke2.io/backup_restore#creating-snapshots). Note that with RKE2, snapshots are stored on each etcd node. This varies from RKE1 which only stores one snapshot per cluster.
#### Metrics
##### Metrics
Option to choose whether to expose etcd metrics to the public or only within the cluster.
### Networking
#### Networking
#### Cluster CIDR
##### Cluster CIDR
IPv4 and/or IPv6 network CIDRs to use for pod IPs (default: 10.42.0.0/16).
##### Dual-stack Networking
###### Dual-stack Networking
To configure [dual-stack](https://docs.rke2.io/install/network_options#dual-stack-configuration) mode, enter a valid IPv4/IPv6 CIDR. For example `10.42.0.0/16,2001:cafe:42:0::/56`.
[Additional configuration](#dual-stack-additional-config) is required when using `cilium` or `multus,cilium` as your [container network](#container-network-provider) interface provider.
#### Service CIDR
##### Service CIDR
IPv4/IPv6 network CIDRs to use for service IPs (default: 10.43.0.0/16).
##### Dual-stack Networking
###### Dual-stack Networking
To configure [dual-stack](https://docs.rke2.io/install/network_options#dual-stack-configuration) mode, enter a valid IPv4/IPv6 CIDR. For example `10.42.0.0/16,2001:cafe:42:0::/56`.
[Additional configuration](#dual-stack-additional-config) is required when using `cilium ` or `multus,cilium` as your [container network](#container-network-provider) interface provider.
#### Cluster DNS
##### Cluster DNS
IPv4 Cluster IP for coredns service. Should be in your service-cidr range (default: 10.43.0.10).
#### Cluster Domain
##### Cluster Domain
Select the domain for the cluster. The default is `cluster.local`.
#### NodePort Service Port Range
##### NodePort Service Port Range
Option to change the range of ports that can be used for [NodePort services](https://kubernetes.io/docs/concepts/services-networking/service/#nodeport). The default is `30000-32767`.
#### Truncate Hostnames
##### Truncate Hostnames
Option to truncate hostnames to 15 characters or less. You can only set this field during the initial creation of the cluster. You can't enable or disable the 15 character limit after cluster creation.
@@ -218,11 +223,11 @@ This setting only affects machine-provisioned clusters. Since custom clusters se
Truncating hostnames in a cluster improves compatibility with Windows-based systems. Although Kubernetes allows hostnames up to 63 characters in length, systems that use NetBIOS restrict hostnames to 15 characters or less.
#### TLS Alternate Names
##### TLS Alternate Names
Add additional hostnames or IPv4/IPv6 addresses as Subject Alternative Names on the server TLS cert.
#### Authorized Cluster Endpoint
##### Authorized Cluster Endpoint
Authorized Cluster Endpoint can be used to directly access the Kubernetes API server, without requiring communication through Rancher.
@@ -232,35 +237,35 @@ For more detail on how an authorized cluster endpoint works and why it is used,
We recommend using a load balancer with the authorized cluster endpoint. For details, refer to the [recommended architecture section.](../../rancher-manager-architecture/architecture-recommendations.md#architecture-for-an-authorized-cluster-endpoint-ace)
### Registries
#### Registries
Select the image repository to pull Rancher images from. For more details and configuration options, see the [RKE2 documentation](https://docs.rke2.io/install/containerd_registry_configuration).
### Upgrade Strategy
#### Upgrade Strategy
#### Control Plane Concurrency
##### Control Plane Concurrency
Select how many nodes can be upgraded at the same time. Can be a fixed number or percentage.
#### Worker Concurrency
##### Worker Concurrency
Select how many nodes can be upgraded at the same time. Can be a fixed number or percentage.
#### Drain Nodes (Control Plane)
##### Drain Nodes (Control Plane)
Option to remove all pods from the node prior to upgrading.
#### Drain Nodes (Worker Nodes)
##### Drain Nodes (Worker Nodes)
Option to remove all pods from the node prior to upgrading.
### Advanced
#### Advanced
Option to set kubelet options for different nodes. For available options, refer to the [Kubernetes documentation](https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/).
## Cluster Config File Reference
Instead of using the Rancher UI to choose Kubernetes options for the cluster, advanced users can create a config file. Using a config file allows you to set the [options available](https://docs.rke2.io/install/configuration) in an RKE2 installation, including those already listed in [Configuration Options in the Rancher UI](#configuration-options-in-the-rancher-ui), as well as Rancher-specific parameters.
Editing clusters in YAML allows you to set the [options available](https://docs.rke2.io/install/configuration) in an RKE2 installation, including those already listed in [Configuration Options in the Rancher UI](#configuration-options-in-the-rancher-ui), as well as set Rancher-specific parameters.
<details>
<summary>
@@ -268,9 +273,11 @@ Instead of using the Rancher UI to choose Kubernetes options for the cluster, ad
</summary>
```yaml
apiVersion: provisioning.cattle.io/v1
kind: Cluster
spec:
cloudCredentialSecretName: cattle-global-data:cc-s879v
kubernetesVersion: v1.23.6+rke2r2
kubernetesVersion: v1.25.12+rke2r1
localClusterAuthEndpoint: {}
rkeConfig:
chartValues:
@@ -283,6 +290,9 @@ spec:
disable-kube-proxy: false
etcd-expose-metrics: false
profile: null
kube-apiserver-arg:
- audit-policy-file=/etc/rancher/rke2/user-audit-policy.yaml
- audit-log-path=/etc/rancher/rke2/user-audit.logs
machinePools:
- controlPlaneRole: true
etcdRole: true
@@ -296,6 +306,18 @@ spec:
machineSelectorConfig:
- config:
protect-kernel-defaults: false
machineSelectorFiles:
- fileSources:
- configMap:
name: ''
secret:
name: audit-policy
items:
- key: audit-policy
path: /etc/rancher/rke2/user-audit-policy.yaml
machineLabelSelector:
matchLabels:
rke.cattle.io/control-plane-role: 'true'
registries: {}
upgradeStrategy:
controlPlaneConcurrency: "1"
@@ -317,7 +339,7 @@ spec:
### chartValues
Option to specify the values for the system charts installed by RKE2/k3s.
Specify the values for the system charts installed by RKE2.
Example:
@@ -328,7 +350,7 @@ chartValues:
```
### machineGlobalConfig
The RKE2/K3s configurations are nested under the `machineGlobalConfig` directive. Any configuration change made here will apply to every node. The configuration options available in the [standalone version of RKE2](https://docs.rke2.io/reference/server_config) can be applied here.
Specify RKE2 configurations. Any configuration change made here will apply to every node. The configuration options available in the [standalone version of RKE2](https://docs.rke2.io/reference/server_config) can be applied here.
Example:
@@ -341,9 +363,9 @@ machineGlobalConfig:
### machineSelectorConfig
This is the same as [`machineGlobalConfig`](#machineglobalconfig) except that a [label](#kubernetes-node-labels) selector can be specified with the configuration. The configuration will only be applied to nodes that match the provided label selector.
`machineSelectorConfig` is the same as [`machineGlobalConfig`](#machineglobalconfig) except that a [label](#kubernetes-node-labels) selector can be specified with the configuration. The configuration will only be applied to nodes that match the provided label selector.
Multiple `config` entries are allowed, each specifying their own `machineLabelSelector`. A user can specify `matchExpressions`, `matchLabels`, both, or neither. Omitting the `machineLabelSelector` section of this has the same effect as putting the config in the `machineGlobalConfig` section.
Multiple `config` entries are allowed, each specifying their own `machineLabelSelector`. A user can specify `matchExpressions`, `matchLabels`, both, or neither. Omitting the `machineLabelSelector` section of this field has the same effect as putting the config in the `machineGlobalConfig` section.
Example:
@@ -362,3 +384,81 @@ machineSelectorConfig
key1: value1
key2: value2
```
### machineSelectorFiles
:::note
This feature is available in Rancher v2.7.2 and later.
:::
Deliver files to nodes, so that the files can be in place before initiating RKE2 server or agent processes.
The content of the file is retrieved from either a secret or a configmap. The target nodes are filtered by the `machineLabelSelector`.
Example :
```yaml
machineSelectorFiles:
- fileSources:
- secret:
items:
- key: example-key
path: path-to-put-the-file-on-nodes
permissions: 644 (optional)
hash: base64-encoded-hash-of-the-content (optional)
name: example-secret-name
machineLabelSelector:
matchExpressions:
- key: example-key
operator: string # Valid operators are In, NotIn, Exists and DoesNotExist.
values:
- example-value1
- example-value2
matchLabels:
key1: value1
key2: value2
- fileSources:
- configMap:
items:
- key: example-key
path: path-to-put-the-file-on-nodes
permissions: 644 (optional)
hash: base64-encoded-hash-of-the-content (optional)
name: example-configmap-name
machineLabelSelector:
matchExpressions:
- key: example-key
operator: string # Valid operators are In, NotIn, Exists and DoesNotExist.
values:
- example-value1
- example-value2
matchLabels:
key1: value1
key2: value2
```
The secret or configmap must meet the following requirements:
1. It must be in the `fleet-default` namespace where the Cluster object exists.
2. It must have the annotation `rke.cattle.io/object-authorized-for-clusters: cluster-name1,cluster-name2`, which permits the target clusters to use it.
:::tip
Rancher Dashboard provides an easy-to-use form for creating the secret or configmap.
:::
Example:
```yaml
apiVersion: v1
data:
audit-policy: >-
IyBMb2cgYWxsIHJlcXVlc3RzIGF0IHRoZSBNZXRhZGF0YSBsZXZlbC4KYXBpVmVyc2lvbjogYXVkaXQuazhzLmlvL3YxCmtpbmQ6IFBvbGljeQpydWxlczoKLSBsZXZlbDogTWV0YWRhdGE=
kind: Secret
metadata:
annotations:
rke.cattle.io/object-authorized-for-clusters: cluster1
name: name1
namespace: fleet-default
```