diff --git a/content/rancher/v2.x/en/admin-settings/feature-flags/_index.md b/content/rancher/v2.x/en/admin-settings/feature-flags/_index.md index f1965430eaf..97baf7a4249 100644 --- a/content/rancher/v2.x/en/admin-settings/feature-flags/_index.md +++ b/content/rancher/v2.x/en/admin-settings/feature-flags/_index.md @@ -6,19 +6,20 @@ _Available as of v2.3.0_ Rancher includes some features that are experimental and disabled by default. You might want to enable these features, for example, if you decide that the benefits of using an [unsupported storage type]({{}}/rancher/v2.x/en/admin-settings/feature-flags/enable-not-default-storage-drivers) outweighs the risk of using an untested feature. Feature flags were introduced to allow you to try these features that are not enabled by default. -The features can be enabled in two ways: +The features can be enabled in three ways: -- When installing Rancher with a CLI, you can use a feature flag to enable a feature by default -- After installing Rancher, you can turn on the features with the Rancher API +- [Enable features when starting Rancher.](#enabling-features-when-starting-rancher) When installing Rancher with a CLI, you can use a feature flag to enable a feature by default. +- [Enable features from the Rancher UI](#enabling-features-with-the-rancher-ui) in Rancher v2.3.3+ by going to the **Settings** page. +- [Enable features with the Rancher API](#enabling-features-with-the-rancher-api) after installing Rancher. Each feature has two values: - A default value, which can be configured with a flag or environment variable from the command line -- A set value, which can be configured with the Rancher API +- A set value, which can be configured with the Rancher API or UI If no value has been set, Rancher uses the default value. -Because the API sets the actual value and the command line sets the default value, that means that if you enable or disable a feature with the API, it will override any value set with the command line. +Because the API sets the actual value and the command line sets the default value, that means that if you enable or disable a feature with the API or UI, it will override any value set with the command line. For example, if you install Rancher, then set a feature flag to true with the Rancher API, then upgrade Rancher with a command that sets the feature flag to false, the default value will still be false, but the feature will still be enabled because it was set with the Rancher API. If you then deleted the set value (true) with the Rancher API, setting it to NULL, the default value (false) would take effect. @@ -82,6 +83,25 @@ docker run -d -p 80:80 -p 443:443 \ {{% /tab %}} {{% /tabs %}} +# Enabling Features with the Rancher UI + +_Available as of Rancher v2.3.3_ + +1. Go to the **Global** view and click **Settings.** +1. Click the **Feature Flags** tab. You will see a list of experimental features. +1. To enable a feature, go to the disabled feature you want to enable and click **Ellipsis (...) > Activate.** + +**Result:** The feature is enabled. + +### Disabling Features with the Rancher UI + +1. Go to the **Global** view and click **Settings.** +1. Click the **Feature Flags** tab. You will see a list of experimental features. +1. To disable a feature, go to the enabled feature you want to disable and click **Ellipsis (...) > Deactivate.** + +**Result:** The feature is disabled. + + # Enabling Features with the Rancher API 1. Go to `/v3/features`. @@ -94,7 +114,7 @@ docker run -d -p 80:80 -p 443:443 \ **Result:** The feature is enabled. -# Disabling Features with the Rancher API +### Disabling Features with the Rancher API 1. Go to `/v3/features`. 1. In the `data` section, you will see an array containing all of the features that can be turned on with feature flags. The name of the feature is in the `id` field. Click the name of the feature you want to enable. @@ -104,4 +124,4 @@ docker run -d -p 80:80 -p 443:443 \ 1. Click **Send Request.** 1. Click **Close.** -**Result:** The feature is disabled. +**Result:** The feature is disabled. \ No newline at end of file diff --git a/content/rancher/v2.x/en/admin-settings/rke-templates/_index.md b/content/rancher/v2.x/en/admin-settings/rke-templates/_index.md index 6ebcedc3ac5..34ac570d8e1 100644 --- a/content/rancher/v2.x/en/admin-settings/rke-templates/_index.md +++ b/content/rancher/v2.x/en/admin-settings/rke-templates/_index.md @@ -17,7 +17,9 @@ Admins control which cluster options can be changed by end users. RKE templates If a cluster was created with an RKE template, you can't change it to a different RKE template. You can only update the cluster to a new revision of the same template. -To summarize, RKE templates allow DevOps and security teams to: +As of Rancher v2.3.3, you can [save the configuration of an existing cluster as an RKE template.]({{}}/rancher/v2.x/en/admin-settings/rke-templates/applying-templates/#converting-an-existing-cluster-to-use-an-rke-template) Then the cluster's settings can only be changed if the template is updated. The new template can also be used to launch new clusters. + +The core features of RKE templates allow DevOps and security teams to: - Standardize cluster configuration and ensure that Rancher-provisioned clusters are created following best practices - Prevent less technical users from making uninformed choices when provisioning clusters @@ -48,7 +50,10 @@ RKE templates are supported for Rancher-provisioned clusters. The templates can RKE templates are for defining Kubernetes and Rancher settings. Node templates are responsible for configuring nodes. For tips on how to use RKE templates in conjunction with hardware, refer to [RKE Templates and Hardware]({{}}/rancher/v2.x/en/admin-settings/rke-templates/rke-templates-and-hardware). -RKE templates can be applied to new clusters, but not existing clusters. +RKE templates can be created from scratch to pre-define cluster configuration. They can be applied to launch new clusters, or templates can also be exported from existing running clusters. + +As of v2.3.3, the settings of an existing cluster can be [saved as an RKE template.]({{}}/rancher/v2.x/en/admin-settings/rke-templates/applying-templates/#converting-an-existing-cluster-to-use-an-rke-template) This creates a new template and binds the cluster settings to the template, so that the cluster can only be upgraded if the [template is updated]({{}}/rancher/v2.x/en/admin-settings/rke-templates/creating-and-revising/#updating-a-template), and the cluster is upgraded to [use a newer version of the template.]({{}}/rancher/v2.x/en/admin-settings/rke-templates/creating-and-revising/#upgrading-a-cluster-to-use-a-new-template-revision) The new template can also be used to create new clusters. + # Example Scenarios When an organization has both basic and advanced Rancher users, administrators might want to give the advanced users more options for cluster creation, while restricting the options for basic users. @@ -58,13 +63,13 @@ These [example scenarios]({{}}/rancher/v2.x/en/admin-settings/rke-templ Some of the example scenarios include the following: - **Enforcing templates:** Administrators might want to [enforce one or more template settings for everyone]({{}}/rancher/v2.x/en/admin-settings/rke-templates/example-scenarios/#enforcing-a-template-setting-for-everyone) if they want all new Rancher-provisioned clusters to have those settings. -- **Sharing different templates with different users:** Administrators might give [different templates to basic and advanced users,]({{}}/rancher/v2.x/en/admin-settings/rke-templates/example-scenarios/#templates-for-basic-and-advanced-users) so that basic users can have more restricted options and advanced users can have more discretion when creating clusters. -- **Updating template settings:** If an organization's security and DevOps teams decide to embed best practices into the required settings for new clusters, those best practices could change over time. If the best practices change, [a template can be updated to a new revision]({{}}/rancher/v2.x/en/admin-settings/rke-templates/example-scenarios/#updating-templates-and-clusters-created-with-them) and clusters created from the template can upgrade to the new version of the template. +- **Sharing different templates with different users:** Administrators might give [different templates to basic and advanced users,]({{}}/rancher/v2.x/en/admin-settings/rke-templates/example-scenarios/#templates-for-basic-and-advanced-users) so that basic users can have more restricted options and advanced users can use more discretion when creating clusters. +- **Updating template settings:** If an organization's security and DevOps teams decide to embed best practices into the required settings for new clusters, those best practices could change over time. If the best practices change, [a template can be updated to a new revision]({{}}/rancher/v2.x/en/admin-settings/rke-templates/example-scenarios/#updating-templates-and-clusters-created-with-them) and clusters created from the template can [upgrade to the new version]({{}}/rancher/v2.x/en/admin-settings/rke-templates/creating-and-revising/#upgrading-a-cluster-to-use-a-new-template-revision) of the template. - **Sharing ownership of a template:** When a template owner no longer wants to maintain a template, or wants to share ownership of the template, this scenario describes how [template ownership can be shared.]({{}}/rancher/v2.x/en/admin-settings/rke-templates/example-scenarios/#allowing-other-users-to-control-and-share-a-template) # Template Management -When you create a RKE template, it is available in the Rancher UI from the **Global** view under **Tools > RKE Templates.** When you create a template, you become the template owner, which gives you permission to revise and share the template. You can share the RKE templates with specific users or groups, and you can also make it public. +When you create an RKE template, it is available in the Rancher UI from the **Global** view under **Tools > RKE Templates.** When you create a template, you become the template owner, which gives you permission to revise and share the template. You can share the RKE templates with specific users or groups, and you can also make it public. Administrators can turn on template enforcement to require users to always use RKE templates when creating a cluster. This allows administrators to guarantee that Rancher always provisions clusters with specific settings. @@ -91,9 +96,12 @@ You can [create a cluster from a template]({{}}/rancher/v2.x/en/admin-s If the RKE template owner creates a new revision of the template, you can [upgrade your cluster to that revision.]({{}}/rancher/v2.x/en/admin-settings/rke-templates/applying-templates/#updating-a-cluster-created-with-an-rke-template) -RKE templates can only be applied to new clusters, not existing clusters. +RKE templates can be created from scratch to pre-define cluster configuration. They can be applied to launch new clusters, or templates can also be exported from existing running clusters. + +As of Rancher v2.3.3, you can [save the configuration of an existing cluster as an RKE template.]({{}}/rancher/v2.x/en/admin-settings/rke-templates/applying-templates/#converting-an-existing-cluster-to-use-an-rke-template) Then the cluster's settings can only be changed if the template is updated. # Standardizing Hardware + RKE templates are designed to standardize Kubernetes and Rancher settings. If you want to standardize your infrastructure as well, you use RKE templates [in conjuction with other tools]({{}}/rancher/v2.x/en/admin-settings/rke-templates/rke-templates-and-hardware). # YAML Customization diff --git a/content/rancher/v2.x/en/admin-settings/rke-templates/applying-templates/_index.md b/content/rancher/v2.x/en/admin-settings/rke-templates/applying-templates/_index.md index 68a5a8d29a5..1a3010aefe0 100644 --- a/content/rancher/v2.x/en/admin-settings/rke-templates/applying-templates/_index.md +++ b/content/rancher/v2.x/en/admin-settings/rke-templates/applying-templates/_index.md @@ -5,9 +5,17 @@ weight: 50 You can create a cluster from an RKE template that you created, or from a template that has been [shared with you.]({{}}/rancher/v2.x/en/admin-settings/rke-templates/template-access-and-sharing) -RKE templates can only be applied to new clusters, not existing clusters. +RKE templates can be applied to new clusters. -You can't change the cluster to use a different RKE template. You can only update the cluster to a new revision of the same template. +As of Rancher v2.3.3, you can [save the configuration of an existing cluster as an RKE template.](#converting-an-existing-cluster-to-use-an-rke-template) Then the cluster's settings can only be changed if the template is updated. + +You can't change a cluster to use a different RKE template. You can only update the cluster to a new revision of the same template. + +This section covers the following topics: + +- [Creating a cluster from an RKE template](#creating-a-cluster-from-an-rke-template) +- [Updating a cluster created with an RKE template](#updating-a-cluster-created-with-an-rke-template) +- [Converting an existing cluster to use an RKE template](#converting-an-existing-cluster-to-use-an-rke-template) ### Creating a Cluster from an RKE Template @@ -30,4 +38,26 @@ When the template owner creates a template, each setting has a switch in the Ran If a cluster was created from an RKE template, you can edit the cluster to update the cluster to a new revision of the template. +As of Rancher v2.3.3, an existing cluster's settings can be [saved as an RKE template.](#converting-an-existing-cluster-to-use-an-rke-template) In that situation, you can also edit the cluster to update the cluster to a new revision of the template. + > **Note:** You can't change the cluster to use a different RKE template. You can only update the cluster to a new revision of the same template. + +### Converting an Existing Cluster to Use an RKE Template + +_Available as of v2.3.3_ + +This section describes how to create an RKE template from an existing cluster. + +RKE templates cannot be applied to existing clusters, except if you save an existing cluster's settings as an RKE template. This exports the cluster's settings as a new RKE template, and also binds the cluster to that template. The result is that the cluster can only be changed if the [template is updated,]({{}}/rancher/v2.x/en/admin-settings/rke-templates/creating-and-revising/#updating-a-template) and the cluster is upgraded to [use a newer version of the template.]({{}}/rancher/v2.x/en/admin-settings/rke-templates/creating-and-revising/#upgrading-a-cluster-to-use-a-new-template-revision) + +To convert an existing cluster to use an RKE template, + +1. From the **Global** view in Rancher, click the **Clusters** tab. +1. Go to the cluster that will be converted to use an RKE template. Click **Ellipsis (...)** > **Save as RKE Template.** +1. Enter a name for the template in the form that appears, and click **Create.** + +**Results:** + +- A new RKE template is created. +- The cluster is converted to use the new template. +- New clusters can be [created from the new template.]({{}}/rancher/v2.x/en/admin-settings/rke-templates/applying-templates/#creating-a-cluster-from-an-rke-template) \ No newline at end of file diff --git a/content/rancher/v2.x/en/admin-settings/rke-templates/creating-and-revising/_index.md b/content/rancher/v2.x/en/admin-settings/rke-templates/creating-and-revising/_index.md index 983aa6bc4f6..1c0c4711596 100644 --- a/content/rancher/v2.x/en/admin-settings/rke-templates/creating-and-revising/_index.md +++ b/content/rancher/v2.x/en/admin-settings/rke-templates/creating-and-revising/_index.md @@ -7,10 +7,25 @@ This section describes how to manage RKE templates and revisions. You an create, Template updates are handled through a revision system. When template owners want to change or update a template, they create a new revision of the template. Individual revisions cannot be edited. However, if you want to prevent a revision from being used to create a new cluster, you can disable it. -Template revisions can be used in two ways: to create a new cluster, or to upgrade a cluster that was created with an earlier version of the template. The template creator can choose a default revision, but when end users create a cluster, they can choose any template and any template revision that is available to them. After the cluster is created from a specific revision, it cannot change to another template, but the cluster can be upgraded to another available revision of the same template. +Template revisions can be used in two ways: to create a new cluster, or to upgrade a cluster that was created with an earlier version of the template. The template creator can choose a default revision, but when end users create a cluster, they can choose any template and any template revision that is available to them. After the cluster is created from a specific revision, it cannot change to another template, but the cluster can be upgraded to a newer available revision of the same template. The template owner has full control over template revisions, and can create new revisions to update the template, delete or disable revisions that should not be used to create clusters, and choose which template revision is the default. +This section covers the following topics: + +- [Prerequisites](#prerequisites) +- [Creating a template](#creating-a-template) +- [Updating a template](#updating-a-template) +- [Deleting a template](#deleting-a-template) +- [Creating a revision based on the default revision](#creating-a-revision-based-on-the-default-revision) +- [Creating a revision based on a cloned revision](#creating-a-revision-based-on-a-cloned-revision) +- [Disabling a template revision](#disabling-a-template-revision) +- [Re-enabling a disabled template revision](#re-enabling-a-disabled-template-revision) +- [Setting a template revision as default](#setting-a-template-revision-as-default) +- [Deleting a template revision](#deleting-a-template-revision) +- [Upgrading a cluster to use a new template revision](#upgrading-a-cluster-to-use-a-new-template-revision) +- [Exporting a running cluster to a new RKE template and revision](#exporting-a-running-cluster-to-a-new-rke-template-and-revision) + ### Prerequisites You can create RKE templates if you have the **Create RKE Templates** permission, which can be [given by an administrator.]({{}}/rancher/v2.x/en/admin-settings/rke-templates/creator-permissions) @@ -25,7 +40,7 @@ You can revise, share, and delete a template if you are an owner of the template 1. Optional: Share the template with other users or groups by [adding them as members.]({{}}/rancher/v2.x/en/admin-settings/rke-templates/template-access-and-sharing/#sharing-templates-with-specific-users) You can also make the template public to share with everyone in the Rancher setup. 1. Then follow the form on screen to save the cluster configuration parameters as part of the template's revision. The revision can be marked as default for this template. -**Result:** An RKE template with one revision is configured. You can use this RKE template revision later when you [provision a Rancher-launched cluster]({{< baseurl >}}/rancher/v2.x/en/cluster-provisioning/rke-clusters). +**Result:** An RKE template with one revision is configured. You can use this RKE template revision later when you [provision a Rancher-launched cluster]({{}}/rancher/v2.x/en/cluster-provisioning/rke-clusters). ### Updating a Template @@ -33,14 +48,14 @@ When you update an RKE template, you are creating a revision of the existing tem You can't edit individual revisions. Since you can't edit individual revisions of a template, in order to prevent a revision from being used, you can [disable it.](#disabling-a-template-revision) -New template revisions can be created without affecting clusters already using a revision of the template. +When new template revisions are created, clusters using an older revision of the template are unaffected. 1. From the **Global** view, click **Tools > RKE Templates.** 1. Go to the template that you want to edit and click the **Vertical Ellipsis (...) > Edit.** 1. Edit the required information and click **Save.** 1. Optional: You can change the default revision of this template and also change who it is shared with. -**Result:** The template is updated. +**Result:** The template is updated. To apply it to a cluster using an older version of the template, refer to the section on [upgrading a cluster to use a new revision of a template.](#upgrading-a-cluster-to-use-a-new-template-revision) ### Deleting a Template @@ -112,4 +127,36 @@ To permanently delete a revision, 1. From the **Global** view, click **Tools > RKE Templates.** 1. Go to the RKE template revision that should be deleted and click the **Ellipsis (...) > Delete.** -**Result:** The RKE template revision is deleted. \ No newline at end of file +**Result:** The RKE template revision is deleted. + +### Upgrading a Cluster to Use a New Template Revision + +> This section assumes that you already have a cluster that [has an RKE template applied.]({{}}/rancher/v2.x/en/admin-settings/rke-templates/applying-templates) +> This section also assumes that you have [updated the template that the cluster is using](#updating-a-template) so that a new template revision is available. + +To upgrade a cluster to use a new template revision, + +1. From the **Global** view in Rancher, click the **Clusters** tab. +1. Go to the cluster that you want to upgrade and click **Ellipsis (...) > Edit.** +1. In the **Cluster Options** section, click the dropdown menu for the template revision, then select the new template revision. +1. Click **Save.** + +**Result:** The cluster is upgraded to use the settings defined in the new template revision. + +### Exporting a Running Cluster to a New RKE Template and Revision + +You can save an existing cluster's settings as an RKE template. + +This exports the cluster's settings as a new RKE template, and also binds the cluster to that template. The result is that the cluster can only be changed if the [template is updated,]({{}}/rancher/v2.x/en/admin-settings/rke-templates/creating-and-revising/#updating-a-template) and the cluster is upgraded to [use a newer version of the template.] + +To convert an existing cluster to use an RKE template, + +1. From the **Global** view in Rancher, click the **Clusters** tab. +1. Go to the cluster that will be converted to use an RKE template. Click **Ellipsis (...)** > **Save as RKE Template.** +1. Enter a name for the template in the form that appears, and click **Create.** + +**Results:** + +- A new RKE template is created. +- The cluster is converted to use the new template. +- New clusters can be [created from the new template and revision.]({{}}/rancher/v2.x/en/admin-settings/rke-templates/applying-templates/#creating-a-cluster-from-an-rke-template) \ No newline at end of file diff --git a/content/rancher/v2.x/en/cluster-provisioning/rke-clusters/node-pools/_index.md b/content/rancher/v2.x/en/cluster-provisioning/rke-clusters/node-pools/_index.md index 7c44a74833f..7fc3ae981df 100644 --- a/content/rancher/v2.x/en/cluster-provisioning/rke-clusters/node-pools/_index.md +++ b/content/rancher/v2.x/en/cluster-provisioning/rke-clusters/node-pools/_index.md @@ -45,6 +45,8 @@ _Available as of Rancher v2.3.0_ If a node is in a node pool, Rancher can automatically replace unreachable nodes. Rancher will use the existing node template for the given node pool to recreate the node if it becomes inactive for a specified number of minutes. +> **Important** Self-healing node pools are designed to help you replace worker nodes for stateless applications. It is not recommended to enable node auto-replace on a node pool of master nodes or nodes with persistent volumes attached, because VMs are treated ephemerally. When a node in a node pool loses connectivity with the cluster, its persistent volumes are destroyed, resulting in data loss for stateful applications. + {{% accordion id="how-does-node-auto-replace-work" label="How does Node Auto-replace Work?" %}} Node auto-replace works on top of the Kubernetes node controller. The node controller periodically checks the status of all the nodes (configurable via the `--node-monitor-period` flag of the `kube-controller`). When a node is unreachable, the node controller will taint that node. When this occurs, Rancher will begin its deletion countdown. You can configure the amount of time Rancher waits to delete the node. If the taint is not removed before the deletion countdown ends, Rancher will proceed to delete the node object. Rancher will then provision a node in accordance with the set quantity of the node pool. {{% /accordion %}} diff --git a/content/rancher/v2.x/en/cluster-provisioning/rke-clusters/node-pools/vsphere/_index.md b/content/rancher/v2.x/en/cluster-provisioning/rke-clusters/node-pools/vsphere/_index.md index bd25eb19a69..7702ec28294 100644 --- a/content/rancher/v2.x/en/cluster-provisioning/rke-clusters/node-pools/vsphere/_index.md +++ b/content/rancher/v2.x/en/cluster-provisioning/rke-clusters/node-pools/vsphere/_index.md @@ -5,170 +5,39 @@ weight: 2225 aliases: - /rancher/v2.x/en/tasks/clusters/creating-a-cluster/create-cluster-vsphere/ --- -Use {{< product >}} to create a Kubernetes cluster in vSphere. -## Introduction +By using Rancher with vSphere, you can bring cloud operations on-premises. -When creating a vSphere cluster, Rancher first provisions the specified amount of virtual machines by communicating with the vCenter API. Then it installs Kubernetes on top of them. A vSphere cluster may consist of multiple groups of VMs with distinct properties, such as the amount of memory or the number of vCPUs. This grouping allows for fine-grained control over the sizing of nodes for the data, control, and worker plane respectively. +Rancher can provision nodes in vSphere and install Kubernetes on them. When creating a Kubernetes cluster in vSphere, Rancher first provisions the specified number of virtual machines by communicating with the vCenter API. Then it installs Kubernetes on top of them. ->**Note:** ->The vSphere node driver included in Rancher currently only supports the provisioning of VMs with [RancherOS]({{< baseurl >}}/os/v1.x/en/) as the guest operating system. +A vSphere cluster may consist of multiple groups of VMs with distinct properties, such as the amount of memory or the number of vCPUs. This grouping allows for fine-grained control over the sizing of nodes for each Kubernetes role. -## Prerequisites +# vSphere Enhancements -### vSphere API permissions +The vSphere node templates have been updated, allowing you to bring cloud operations on-premises with the following enhancements: -Before proceeding to create a cluster, you must ensure that you have a vSphere user with sufficient permissions. If you are planning to make use of vSphere volumes for persistent storage in the cluster, there are [additional requirements]({{< baseurl >}}/rke/latest/en/config-options/cloud-providers/vsphere/) that must be met. +### Self-healing Node Pools -### Network permissions +_Available as of v2.3.0_ -You must ensure that the hosts running Rancher servers are able to establish network connections to the following network endpoints: +One of the biggest advantages of provisioning vSphere nodes with Rancher is that it allows you to take advantage of Rancher's self-healing node pools, also called the [node auto-replace feature,]({{}}/rancher/v2.x/en/cluster-provisioning/rke-clusters/node-pools/#node-auto-replace) in your on-premises clusters. Self-healing node pools are designed to help you replace worker nodes for stateless applications. When Rancher provisions nodes from a node template, Rancher can automatically replace unreachable nodes. -- vCenter server (usually port 443/TCP) -- Every ESXi host that is part of the datacenter to be used to provision virtual machines for your clusters (port 443/TCP). +> **Important:** It is not recommended to enable node auto-replace on a node pool of master nodes or nodes with persistent volumes attached, because VMs are treated ephemerally. When a node in a node pool loses connectivity with the cluster, its persistent volumes are destroyed, resulting in data loss for stateful applications. +### Dynamically Populated Options for Instances and Scheduling -## Provisioning a vSphere Cluster +_Available as of v2.3.3_ -The following steps create a role with the required privileges and then assign it to a new user in the vSphere console: +Node templates for vSphere have been updated so that when you create a node template with your vSphere credentials, the template is automatically populated with the same options for provisioning VMs that you have access to in the vSphere console. -1. From the **vSphere** console, go to the **Administration** page. +For the fields to be populated, your setup needs to fulfill the [prerequisites.]({{}}/rancher/v2.x/en/cluster-provisioning/rke-clusters/node-pools/vsphere/provisioning-vsphere-clusters/#prerequisites) -2. Go to the **Roles** tab. +### More Supported Operating Systems -3. Create a new role. Give it a name and select the privileges listed in the [permissions table](#annex-vsphere-permissions). +As of Rancher v2.3.3+, you can provision VMs with any operating system that supports cloud init. - {{< img "/img/rancher/rancherroles1.png" "image">}} +In Rancher prior to v2.3.3, the vSphere node driver included in Rancher only supported the provisioning of VMs with [RancherOS]({{}}/os/v1.x/en/) as the guest operating system. -4. Go to the **Users and Groups** tab. +# Video Walkthrough of v2.3.3 Node Template Features -5. Create a new user. Fill out the form and then click **OK**. Make sure to note the username and password, as you will need it when configuring node templates in Rancher. - - {{< img "/img/rancher/rancheruser.png" "image">}} - -6. Go to the **Global Permissions** tab. - -7. Create a new Global Permission. Add the user you created earlier and assign it the role you created earlier. Click **OK**. - - {{< img "/img/rancher/globalpermissionuser.png" "image">}} - - {{< img "/img/rancher/globalpermissionrole.png" "image">}} - -## Creating vSphere Clusters - -### Create a vSphere Node Template - -To create a cluster, you need to create at least one vSphere [node template]({{< baseurl >}}/rancher/v2.x/en/cluster-provisioning/rke-clusters/node-pools/#node-templates) that specifies how VMs are created in vSphere. - ->**Note:** ->Once you create a node template, it is saved, and you can re-use it whenever you create additional vSphere clusters. - -1. Log in with an admin account to the Rancher UI. - -2. From the user settings menu, select **Node Templates**. - -3. Click **Add Template** and then click on the **vSphere** icon. - -4. Under [Account Access](#account-access) enter the vCenter FQDN or IP address and the credentials for the vSphere user account (see [Prerequisites](#prerequisites)). - - {{< step_create-cloud-credential >}} - -5. Under [Instance Options](#instance-options), configure the number of vCPUs, memory, and disk size for the VMs created by this template. - -6. **Optional:** Enter the URL pointing to a [RancherOS]({{< baseurl >}}/os/v1.x/en/) cloud-config file in the [Cloud Init](#instance-options) field. - -7. Ensure that the [OS ISO URL](#instance-options) contains the URL of a VMware ISO release for RancherOS (`rancheros-vmware.iso`). - - {{< img "/img/rancher/vsphere-node-template-1.png" "image">}} - -8. **Optional:** Provide a set of [Configuration Parameters](#instance-options) for the VMs. - -9. Under **Scheduling**, enter the name/path of the **Data Center** to create the VMs in, the name of the **VM Network** to attach to, and the name/path of the **Datastore** to store the disks in. - - {{< img "/img/rancher/vsphere-node-template-2.png" "image">}} - -10. **Optional:** Assign labels to the VMs that can be used as a base for scheduling rules in the cluster. - -11. **Optional:** Customize the configuration of the Docker daemon on the VMs that will be created. - -10. Assign a descriptive **Name** for this template and click **Create**. - -___ - -### Create a vSphere Cluster - -After you've created a template, you can use it stand up the vSphere cluster itself. - -1. From the **Global** view, click **Add Cluster**. - -2. Choose **vSphere**. - -3. Enter a **Cluster Name**. - -4. {{< step_create-cluster_member-roles >}} - -5. {{< step_create-cluster_cluster-options >}} - -6. {{< step_create-cluster_node-pools >}} - - {{< img "/img/rancher/vsphere-cluster-create-1.png" "Image">}} - -7. Review your configuration, then click **Create**. - -> **Note:** -> -> If you have a cluster with DRS enabled, setting up [VM-VM Affinity Rules](https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.resmgmt.doc/GUID-7297C302-378F-4AF2-9BD6-6EDB1E0A850A.html) is recommended. These rules allow VMs assigned the etcd and control-plane roles to operate on separate ESXi hosts when they are assigned to different node pools. This practice ensures that the failure of a single physical machine does not affect the availability of those planes. - -{{< result_create-cluster >}} - -## Annex - Node Template Configuration Reference - -The tables below describe the configuration options available in the vSphere node template. - -### Account Access - -| Parameter | Required | Description | -|:------------------------:|:--------:|:------------------------------------------------------------:| -| vCenter or ESXi Server | * | IP or FQDN of the vCenter or ESXi server used for managing VMs. | -| Port | * | Port to use when connecting to the server. Defaults to `443`. | -| Username | * | vCenter/ESXi user to authenticate with the server. | -| Password | * | User's password. | - -___ - -### Instance Options - -| Parameter | Required | Description | -|:------------------------:|:--------:|:------------------------------------------------------------:| -| CPUs | * | Number of vCPUS to assign to VMs. | -| Memory | * | Amount of memory to assign to VMs. | -| Disk | * | Size of the disk (in MB) to attach to the VMs. | -| Cloud Init | | URL of a [RancherOS cloud-config]({{< baseurl >}}/os/v1.x/en/installation/configuration/) file to provision VMs with. This file allows further customization of the RancherOS operating system, such as network configuration, DNS servers, or system daemons.| -| OS ISO URL | * | URL of a RancherOS vSphere ISO file to boot the VMs from. You can find URLs for specific versions in the [Rancher OS GitHub Repo](https://github.com/rancher/os). | -| Configuration Parameters | | Additional configuration parameters for the VMs. These correspond to the [Advanced Settings](https://kb.vmware.com/s/article/1016098) in the vSphere console. Example use cases include providing RancherOS [guestinfo]({{< baseurl >}}/os/v1.x/en/installation/running-rancheros/cloud/vmware-esxi/#vmware-guestinfo) parameters or enabling disk UUIDs for the VMs (`disk.EnableUUID=TRUE`). | - -___ - -### Scheduling Options - -| Parameter | Required | Description | -|:------------------------:|:--------:|:------------------------------------------------------------:| -| Data Center | * | Name/path of the datacenter to create VMs in. | -| Pool | | Name/path of the resource pool to schedule the VMs in. If not specified, the default resource pool is used. | -| Host | | Name/path of the host system to schedule VMs in. If specified, the host system's pool will be used and the *Pool* parameter will be ignored. | -| Network | * | Name of the VM network to attach VMs to. | -| Data Store | * | Datastore to store the VM disks. | -| Folder | | Name/path of folder in the datastore to create the VMs in. Must already exist. | - -___ - -## Annex - vSphere Permissions - -The following table lists the permissions required for the vSphere user account configured in the node templates: - -| Privilege Group | Operations | -|:----------------------|:-----------------------------------------------------------------------| -| Datastore | AllocateSpace
Browse
FileManagement (Low level file operations)
UpdateVirtualMachineFiles
UpdateVirtualMachineMetadata | -| Network | Assign | -| Resource | AssignVMToPool | -| Virtual Machine | Config (All)
GuestOperations (All)
Interact (All)
Inventory (All)
Provisioning (All) | +In [this YouTube video,](https://www.youtube.com/watch?v=dPIwg6x1AlU) we demonstrate how to set up a node template with the new features designed to help you bring cloud operations to on-premises clusters. diff --git a/content/rancher/v2.x/en/cluster-provisioning/rke-clusters/node-pools/vsphere/provisioning-vsphere-clusters/_index.md b/content/rancher/v2.x/en/cluster-provisioning/rke-clusters/node-pools/vsphere/provisioning-vsphere-clusters/_index.md new file mode 100644 index 00000000000..39fa35b7528 --- /dev/null +++ b/content/rancher/v2.x/en/cluster-provisioning/rke-clusters/node-pools/vsphere/provisioning-vsphere-clusters/_index.md @@ -0,0 +1,305 @@ +--- +title: Provisioning Kubernetes Clusters in vSphere +weight: 1 +--- + +This section explains how to configure Rancher with vSphere credentials, provision nodes in vSphere, and set up Kubernetes clusters on those nodes. + +# Prerequisites + +This section describes the requirements for setting up vSphere so that Rancher can provision VMs and clusters. + +The node templates are documented and tested with the vSphere Web Services API version 6.5. + +- [Create credentials in vSphere](#create-credentials-in-vsphere) +- [Network permissions](#network-permissions) +- [Valid ESXi License for vSphere API Access](#valid-esxi-license-for-vsphere-api-access) + +### Create Credentials in vSphere + +Before proceeding to create a cluster, you must ensure that you have a vSphere user with sufficient permissions. When you set up a node template, the template will need to use these vSphere credentials. + +Refer to this [how-to guide]({{}}/rancher/v2.x/en/cluster-provisioning/rke-clusters/node-pools/vsphere/provisioning-vsphere-clusters/creating-credentials) for instructions on how to create a user in vSphere with the required permissions. These steps result in a username and password that you will need to provide to Rancher, which allows Rancher to provision resources in vSphere. + +### Network Permissions + +There needs to be two-way communication between Rancher and the vSphere API. + +You must ensure that the hosts running Rancher servers are able to establish network connections to the following network endpoints: + +- vCenter server (usually port 443/TCP) +- Every ESXi host that is part of the datacenter to be used to provision virtual machines for your clusters (port 443/TCP). + +By default, Rancher uses port 443 to communicate with vSphere. + +The vSphere API websocket port will be 84453 by default. + +### Valid ESXi License for vSphere API Access + +The free ESXi license does not support API access. The vSphere servers must have a valid or evaluation ESXi license. + +# Creating Clusters in vSphere with Rancher + +This section describes how to set up vSphere credentials, node templates, and vSphere clusters using the Rancher UI. + +You will need to do the following: + +1. [Create a node template using vSphere credentials](#1-create-a-node-template-using-vsphere-credentials) +2. [Create a Kubernetes cluster using the node template](#2-create-a-kubernetes-cluster-using-the-node-template) +3. [Optional: Provision storage](#3-optional-provision-storage) + - [Enable the vSphere cloud provider for the cluster](#enable-the-vsphere-cloud-provider-for-the-cluster) + +### Configuration References + +For details on configuring the node template, refer to the [node template configuration reference.]({{}}/rancher/v2.x/en/cluster-provisioning/rke-clusters/node-pools/vsphere/provisioning-vsphere-clusters/node-template-reference/) + +Rancher uses the RKE library to provision Kubernetes clusters. For details on configuring clusters in vSphere, refer to the [cluster configuration reference in the RKE documentation.]({{}}/rke/latest/en/config-options/cloud-providers/vsphere/config-reference/) + +Note that the vSphere cloud provider must be [enabled](#enable-the-vsphere-cloud-provider-for-the-cluster) to allow dynamic provisioning of volumes. + +# 1. Create a Node Template Using vSphere Credentials + +To create a cluster, you need to create at least one vSphere [node template]({{}}/rancher/v2.x/en/cluster-provisioning/rke-clusters/node-pools/#node-templates) that specifies how VMs are created in vSphere. + +After you create a node template, it is saved, and you can re-use it whenever you create additional vSphere clusters. + +To create a node template, + +1. Log in with an admin account to the Rancher UI. + +1. From the user settings menu, select **Node Templates.** + +1. Click **Add Template** and then click on the **vSphere** icon. + +Then, configure your template: + +- [A. Configure the vSphere credential](#a-configure-the-vsphere-credential) +- [B. Configure node scheduling](#b-configure-node-scheduling) +- [C. Configure instances and operating systems](#c-configure-instances-and-operating-systems) +- [D. Add networks](#d-add-networks) +- [E. If not already enabled, enable disk UUIDs](#e-if-not-already-enabled-enable-disk-uuids) +- [F. Optional: Configure node tags and custom attributes](#f-optional-configure-node-tags-and-custom-attributes) +- [G. Optional: Configure cloud-init](#g-optional-configure-cloud-init) +- [H. Saving the node template](#h-saving-the-node-template) + +### A. Configure the vSphere Credential + +The steps for configuring your vSphere credentials for the cluster are different depending on your version of Rancher. + +{{% tabs %}} +{{% tab "Rancher v2.2.0+" %}} + +Your account access information is in a [cloud credential.]({{}}/rancher/v2.x/en/user-settings/cloud-credentials/) Cloud credentials are stored as Kubernetes secrets. + +You can use an existing cloud credential or create a new one. To create a new cloud credential, + +1. Click **Add New.** +1. In the **Name** field, enter a name for your vSphere credentials. +1. In the **vCenter or ESXi Server** field, enter the vCenter or ESXi hostname/IP. ESXi is the virtualization platform where you create and run virtual machines and virtual appliances. vCenter Server is the service through which you manage multiple hosts connected in a network and pool host resources. +1. Optional: In the **Port** field, configure the port of the vCenter or ESXi server. +1. In the **Username** and **Password** fields, enter your vSphere login username and password. +1. Click **Create.** + +**Result:** The node template has the credentials required to provision nodes in vSphere. + +{{% /tab %}} +{{% tab "Rancher prior to v2.2.0" %}} +In the **Account Access** section, enter the vCenter FQDN or IP address and the credentials for the vSphere user account. +{{% /tab %}} +{{% /tabs %}} + +### B. Configure Node Scheduling + +Choose what hypervisor the virtual machine will be scheduled to. The configuration options depend on your version of Rancher. + +{{% tabs %}} +{{% tab "Rancher v2.3.3+" %}} + +The fields in the **Scheduling** section should auto-populate with the data center and other scheduling options that are available to you in vSphere. + +1. In the **Data Center** field, choose the data center where the VM will be scheduled. +1. Optional: Select a **Resource Pool.** Resource pools can be used to partition available CPU and memory resources of a standalone host or cluster, and they can also be nested. +1. If you have a data store cluster, you can toggle the **Data Store** field. This lets you select a data store cluster where your VM will be scheduled to. If the field is not toggled, you can select an individual disk. +1. Optional: Select a folder where the VM will be placed. The VM folders in this dropdown menu directly correspond to your VM folders in vSphere. Note: The folder name should be prefaced with `vm/` in your vSphere config file. +1. Optional: Choose a specific host to create the VM on. Leave this field blank for a standalone ESXi or for a cluster with DRS (Distributed Resource Scheduler). If specified, the host system's pool will be used and the **Resource Pool** parameter will be ignored. +{{% /tab %}} +{{% tab "Rancher prior to v2.3.3" %}} + +In the **Scheduling** section, enter: + +- The name/path of the **Data Center** to create the VMs in +- The name of the **VM Network** to attach to +- The name/path of the **Datastore** to store the disks in + + ![image]({{< baseurl >}}/img/rancher/vsphere-node-template-2.png) + +{{% /tab %}} +{{% /tabs %}} + +### C. Configure Instances and Operating Systems + +The instances are configured differently depending on your Rancher version. + +{{% tabs %}} +{{% tab "Rancher v2.3.3+" %}} + +In this section, configure the number of vCPUs, memory, and disk size for the VMs created by this template. + +In the **Creation method** field, you will configure the method for setting up an operating system on the node. The operating system can be installed from an ISO or from a VM template. + +[VM templates](https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.vm_admin.doc/GUID-F7BF0E6B-7C4F-4E46-8BBF-76229AEA7220.html) are useful for setting up the operating system and other software, because they allow you to save time. For example, you could use a VM template to automatically install Kubernetes and Docker on each node. You can choose ISOs defined from templates in a vSphere data center or content library. + +The node can be created with any operating system that supports `cloud-init`. + +Choose the way that the VM will be created: + +- **Deploy from template: Data Center:** Choose a template that exists in the data center that you selected. +- **Deploy from template: Content Library:** In the two fields that appear when you select this option, choose the [content library](https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.vm_admin.doc/GUID-254B2CE8-20A8-43F0-90E8-3F6776C2C896.html). Then select the [VM template](https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.vm_admin.doc/GUID-F7BF0E6B-7C4F-4E46-8BBF-76229AEA7220.html) from the list of templates within the content library. This template will be used to create the new VM. +- **Clone an existing virtual machine:** In the **Virtual machine** field, choose an existing VM that the new VM will be cloned from. +- **Install from boot2docker ISO:** Ensure that the OS ISO URL contains the URL of a VMware ISO release for RancherOS (rancheros-vmware.iso). + +{{% /tab %}} +{{% tab "Rancher prior to v2.3.3" %}} + +In the **Instance Options** section, configure the number of vCPUs, memory, and disk size for the VMs created by this template. + +Only RancherOS VMs are supported. + +Ensure that the [OS ISO URL](#instance-options) contains the URL of the VMware ISO release for RancherOS: `rancheros-vmware.iso`. + + ![image]({{< baseurl >}}/img/rancher/vsphere-node-template-1.png) + +{{% /tab %}} +{{% /tabs %}} + +### D. Add Networks + +_Available as of v2.3.3_ + +The node template now allows a VM to be provisioned with multiple networks. In the **Networks** field, you can now click **Add Network** to add any networks available to you in vSphere. + +### E. If Not Already Enabled, Enable Disk UUIDs + +In order to provision nodes with RKE, all nodes must be configured with disk UUIDs. + +As of Rancher v2.0.4, disk UUIDs are enabled in vSphere node templates by default. + +If you are using Rancher prior to v2.0.4, refer to these [instructions]({{}}/rancher/v2.x/en/cluster-provisioning/rke-clusters/node-pools/vsphere/provisioning-vsphere-clusters/#enabling-disk-uuids-with-a-node-template) for details on how to enable a UUID with a Rancher node template. + +### F. Optional: Configure Node Tags and Custom Attributes + +The way to attach metadata to the VM is different depending on your Rancher version. + +{{% tabs %}} +{{% tab "Rancher v2.3.3+" %}} + +**Optional:** Add vSphere tags and custom attributes. Tags allow you to attach metadata to objects in the vSphere inventory to make it easier to sort and search for these objects. + +For tags, all your vSphere tags will show up as options to select from in your node template. + +In the custom attributes, Rancher will let you select all the custom attributes you have already set up in vSphere. The custom attributes are keys and you can enter values for each one. + + > **Note:** Custom attributes are a legacy feature that will eventually be removed from vSphere. These attributes allow you to attach metadata to objects in the vSphere inventory to make it easier to sort and search for these objects. + +{{% /tab %}} +{{% tab "Rancher prior to v2.3.3" %}} + +**Optional:** + + - Provide a set of configuration parameters (instance-options) for the VMs. + - Assign labels to the VMs that can be used as a base for scheduling rules in the cluster. + - Customize the configuration of the Docker daemon on the VMs that will be created. + +> **Note:** Custom attributes are a legacy feature that will eventually be removed from vSphere. These attributes allow you to attach metadata to objects in the vSphere inventory to make it easier to sort and search for these objects. + +{{% /tab %}} +{{% /tabs %}} + +### G. Optional: Configure Cloud Init + +[Cloud-init](https://cloud-init.io/) is a tool that applies user data to your nodes when they boot for the first time. + +The configuration file for `cloud-init` is named `cloud-config.yml.` In the **Cloud Init** field, it is optional to enter a file name or URL pointing to a `cloud-config.yml` file. + +You can use `cloud-init` to automate tasks that should happen when the instance boots, such as creating users, running shell commands, adding a load balancer, or preinstalling Kubernetes on the VM. + + For examples of how to write a `cloud-config` file, refer to the [cloud-init documentation.](https://cloudinit.readthedocs.io/en/latest/topics/examples.html) + +### H. Saving the Node Template + +Assign a descriptive **Name** for this template and click **Create.** + +### Node Template Configuration Reference + +Refer to [this section]({{}}/rancher/v2.x/en/cluster-provisioning/rke-clusters/node-pools/vsphere/provisioning-vsphere-clusters/node-template-reference/) for a reference on the configuration options available for vSphere node templates. + +# 2. Create a Kubernetes Cluster Using the Node Template + +After you've created a template, you can use it to stand up the vSphere cluster itself. + +To install Kubernetes on vSphere nodes, you will need to enable the vSphere cloud provider by modifying the cluster YAML file. This requirement applies to both pre-created [custom nodes]({{}}/rancher/v2.x/en/cluster-provisioning/rke-clusters/custom-nodes/) and for nodes created in Rancher using the vSphere node driver. + +To create the cluster and enable the vSphere provider for cluster, follow these steps: + +- [A. Set up the cluster name and member roles](#a-set-up-the-cluster-name-and-member-roles) +- [B. Configure Kubernetes options](#b-configure-kubernetes-options) +- [C. Add node pools to the cluster](#c-add-node-pools-to-the-cluster) +- [D. Optional: Add a self-healing node pool](#d-optional-add-a-self-healing-node-pool) +- [E. Create the cluster](#e-create-the-cluster) + +### A. Set up the Cluster Name and Member Roles + +1. Log in to the Rancher UI as an admin user. +2. Navigate to **Clusters** in the **Global** view. +3. Click **Add Cluster** and select the **vSphere** infrastructure provider. +4. Assign a **Cluster Name.** +5. Assign **Member Roles** as required. {{< step_create-cluster_member-roles >}} + +> **Note:** +> +> If you have a cluster with DRS enabled, setting up [VM-VM Affinity Rules](https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.resmgmt.doc/GUID-7297C302-378F-4AF2-9BD6-6EDB1E0A850A.html) is recommended. These rules allow VMs assigned the etcd and control-plane roles to operate on separate ESXi hosts when they are assigned to different node pools. This practice ensures that the failure of a single physical machine does not affect the availability of those planes. + + +### B. Configure Kubernetes Options +{{}} + +### C. Add Node Pools to the Cluster +{{}} + +### D. Optional: Add a Self-Healing Node Pool + +To make a node pool self-healing, enter a number greater than zero in the **Auto Replace** column. Rancher will use the node template for the given node pool to recreate the node if it becomes inactive for that number of minutes. + +> **Note:** Self-healing node pools are designed to help you replace worker nodes for stateless applications. It is not recommended to enable node auto-replace on a node pool of master nodes or nodes with persistent volumes attached, because VMs are treated ephemerally. When a node in a node pool loses connectivity with the cluster, its persistent volumes are destroyed, resulting in data loss for stateful applications. + +### E. Create the Cluster + +Click **Create** to start provisioning the VMs and Kubernetes services. + +{{< result_create-cluster >}} + +# 3. Optional: Provision Storage + +For an example of how to provision storage in vSphere using Rancher, refer to the + [cluster administration section.]({{}}/rancher/v2.x/en/cluster-admin/volumes-and-storage/examples/vsphere) + + In order to provision storage in vSphere, the vSphere provider must be enabled. + +### Enable the vSphere Cloud Provider for the Cluster + +1. Set **Cloud Provider** option to `Custom`. + + ![vsphere-node-driver-cloudprovider]({{< baseurl >}}/img/rancher/vsphere-node-driver-cloudprovider.png) + +1. Click on **Edit as YAML** +1. Insert the following structure to the pre-populated cluster YAML. As of Rancher v2.3+, this structure must be placed under `rancher_kubernetes_engine_config`. In versions prior to v2.3, it has to be defined as a top-level field. Note that the `name` *must* be set to `vsphere`. + + ```yaml + rancher_kubernetes_engine_config: # Required as of Rancher v2.3+ + cloud_provider: + name: vsphere + vsphereCloudProvider: + [Insert provider configuration] + ``` + + Rancher uses RKE (the Rancher Kubernetes Engine) to provision Kubernetes clusters. Refer to the [vSphere configuration reference in the RKE documentation]({{}}/rke/latest/en/config-options/cloud-providers/vsphere/config-reference/) for details about the properties of the `vsphereCloudProvider` directive. diff --git a/content/rancher/v2.x/en/cluster-provisioning/rke-clusters/node-pools/vsphere/provisioning-vsphere-clusters/creating-credentials/_index.md b/content/rancher/v2.x/en/cluster-provisioning/rke-clusters/node-pools/vsphere/provisioning-vsphere-clusters/creating-credentials/_index.md new file mode 100644 index 00000000000..26479897974 --- /dev/null +++ b/content/rancher/v2.x/en/cluster-provisioning/rke-clusters/node-pools/vsphere/provisioning-vsphere-clusters/creating-credentials/_index.md @@ -0,0 +1,41 @@ +--- +title: Creating Credentials in the vSphere Console +weight: 1 +--- + +This section describes how to create a vSphere username and password. You will need to provide these vSphere credentials to Rancher, which allows Rancher to provision resources in vSphere. + +The following table lists the permissions required for the vSphere user account: + +| Privilege Group | Operations | +|:----------------------|:-----------------------------------------------------------------------| +| Datastore | AllocateSpace
Browse
FileManagement (Low level file operations)
UpdateVirtualMachineFiles
UpdateVirtualMachineMetadata | +| Network | Assign | +| Resource | AssignVMToPool | +| Virtual Machine | Config (All)
GuestOperations (All)
Interact (All)
Inventory (All)
Provisioning (All) | + +The following steps create a role with the required privileges and then assign it to a new user in the vSphere console: + +1. From the **vSphere** console, go to the **Administration** page. + +2. Go to the **Roles** tab. + +3. Create a new role. Give it a name and select the privileges listed in the permissions table above. + + ![image]({{< baseurl >}}/img/rancher/rancherroles1.png) + +4. Go to the **Users and Groups** tab. + +5. Create a new user. Fill out the form and then click **OK**. Make sure to note the username and password, because you will need it when configuring node templates in Rancher. + + ![image]({{< baseurl >}}/img/rancher/rancheruser.png) + +6. Go to the **Global Permissions** tab. + +7. Create a new Global Permission. Add the user you created earlier and assign it the role you created earlier. Click **OK**. + + ![image]({{< baseurl >}}/img/rancher/globalpermissionuser.png) + + ![image]({{< baseurl >}}/img/rancher/globalpermissionrole.png) + +**Result:** You now have credentials that Rancher can use to manipulate vSphere resources. \ No newline at end of file diff --git a/content/rancher/v2.x/en/cluster-provisioning/rke-clusters/node-pools/vsphere/provisioning-vsphere-clusters/enabling-uuids/_index.md b/content/rancher/v2.x/en/cluster-provisioning/rke-clusters/node-pools/vsphere/provisioning-vsphere-clusters/enabling-uuids/_index.md new file mode 100644 index 00000000000..4cb5a130602 --- /dev/null +++ b/content/rancher/v2.x/en/cluster-provisioning/rke-clusters/node-pools/vsphere/provisioning-vsphere-clusters/enabling-uuids/_index.md @@ -0,0 +1,24 @@ +--- +title: Enabling Disk UUIDs in Node Templates +weight: 3 +--- + +As of Rancher v2.0.4, disk UUIDs are enabled in vSphere node templates by default. + +For Rancher prior to v2.0.4, we recommend configuring a vSphere node template to automatically enable disk UUIDs because they are required for Rancher to manipulate vSphere resources. + +To enable disk UUIDs for all VMs created for a cluster, + +1. Navigate to the **Node Templates** in the Rancher UI while logged in as admin user. + +2. Add or edit an existing vSphere node template. + +3. Under **Instance Options** click on **Add Parameter**. + +4. Enter `disk.enableUUID` as key with a value of **TRUE**. + + ![vsphere-nodedriver-enable-uuid]({{< baseurl >}}/img/rke/vsphere-nodedriver-enable-uuid.png) + +5. Click **Create** or **Save**. + +**Result:** The disk UUID is enabled in the vSphere node template. \ No newline at end of file diff --git a/content/rancher/v2.x/en/cluster-provisioning/rke-clusters/node-pools/vsphere/provisioning-vsphere-clusters/node-template-reference/_index.md b/content/rancher/v2.x/en/cluster-provisioning/rke-clusters/node-pools/vsphere/provisioning-vsphere-clusters/node-template-reference/_index.md new file mode 100644 index 00000000000..cdc3d70e232 --- /dev/null +++ b/content/rancher/v2.x/en/cluster-provisioning/rke-clusters/node-pools/vsphere/provisioning-vsphere-clusters/node-template-reference/_index.md @@ -0,0 +1,93 @@ +--- +title: vSphere Node Template Configuration Reference +weight: 4 +--- + +The tables below describe the configuration options available in the vSphere node template: + +- [Account access](#account-access) +- [Instance options](#instance-options) +- [Scheduling options](#scheduling-options) + +# Account Access + +The account access parameters are different based on the Rancher version. + +{{% tabs %}} +{{% tab "Rancher v2.2.0+" %}} + +| Parameter | Required | Description | +|:----------------------|:--------:|:-----| +| Cloud Credentials | * | Your vSphere account access information, stored in a [cloud credential.]({{}}/rancher/v2.x/en/user-settings/cloud-credentials/) | + +{{% /tab %}} +{{% tab "Rancher prior to v2.2.0" %}} + +| Parameter | Required | Description | +|:------------------------|:--------:|:------------------------------------------------------------| +| vCenter or ESXi Server | * | IP or FQDN of the vCenter or ESXi server used for managing VMs. | +| Port | * | Port to use when connecting to the server. Defaults to `443`. | +| Username | * | vCenter/ESXi user to authenticate with the server. | +| Password | * | User's password. | + +{{% /tab %}} +{{% /tabs %}} + +# Instance Options + +The options for creating and configuring an instance are different depending on your Rancher version. + +{{% tabs %}} +{{% tab "Rancher v2.3.3+" %}} + +| Parameter | Required | Description | +|:----------------|:--------:|:-----------| +| CPUs | * | Number of vCPUS to assign to VMs. | +| Memory | * | Amount of memory to assign to VMs. | +| Disk | * | Size of the disk (in MB) to attach to the VMs. | +| Creation method | * | The method for setting up an operating system on the node. The operating system can be installed from an ISO or from a VM template. Depending on the creation method, you will also have to specify a VM template, content library, existing VM, or ISO. For more information on creation methods, refer to the section on [configuring instances.]({{}}/rancher/v2.x/en/cluster-provisioning/rke-clusters/node-pools/vsphere/provisioning-vsphere-clusters/#c-configure-instances-and-operating-systems) | +| Cloud Init | | URL of a `cloud-config.yml` file or URL to provision VMs with. This file allows further customization of the operating system, such as network configuration, DNS servers, or system daemons. The operating system must support `cloud-init`. | +| Networks | | Name(s) of the network to attach the VM to. | +| Configuration Parameters used for guestinfo | | Additional configuration parameters for the VMs. These correspond to the [Advanced Settings](https://kb.vmware.com/s/article/1016098) in the vSphere console. Example use cases include providing RancherOS [guestinfo]({{< baseurl >}}/os/v1.x/en/installation/running-rancheros/cloud/vmware-esxi/#vmware-guestinfo) parameters or enabling disk UUIDs for the VMs (`disk.EnableUUID=TRUE`). | + +{{% /tab %}} +{{% tab "Rancher prior to v2.3.3" %}} + +| Parameter | Required | Description | +|:------------------------|:--------:|:------------------------------------------------------------| +| CPUs | * | Number of vCPUS to assign to VMs. | +| Memory | * | Amount of memory to assign to VMs. | +| Disk | * | Size of the disk (in MB) to attach to the VMs. | +| Cloud Init | | URL of a [RancherOS cloud-config]({{< baseurl >}}/os/v1.x/en/installation/configuration/) file to provision VMs with. This file allows further customization of the RancherOS operating system, such as network configuration, DNS servers, or system daemons.| +| OS ISO URL | * | URL of a RancherOS vSphere ISO file to boot the VMs from. You can find URLs for specific versions in the [Rancher OS GitHub Repo](https://github.com/rancher/os). | +| Configuration Parameters | | Additional configuration parameters for the VMs. These correspond to the [Advanced Settings](https://kb.vmware.com/s/article/1016098) in the vSphere console. Example use cases include providing RancherOS [guestinfo]({{< baseurl >}}/os/v1.x/en/installation/running-rancheros/cloud/vmware-esxi/#vmware-guestinfo) parameters or enabling disk UUIDs for the VMs (`disk.EnableUUID=TRUE`). | + +{{% /tab %}} +{{% /tabs %}} + +# Scheduling Options +The options for scheduling VMs to a hypervisor are different depending on your Rancher version. +{{% tabs %}} +{{% tab "Rancher v2.3.3+" %}} + +| Parameter | Required | Description | +|:------------------------|:--------:|:-------| +| Data Center | * | Name/path of the datacenter to create VMs in. | +| Resource Pool | | Name of the resource pool to schedule the VMs in. Leave blank for standalone ESXi. If not specified, the default resource pool is used. | +| Data Store | * | If you have a data store cluster, you can toggle the **Data Store** field. This lets you select a data store cluster where your VM will be scheduled to. If the field is not toggled, you can select an individual disk. | +| Folder | | Name of a folder in the datacenter to create the VMs in. Must already exist. The folder name should be prefaced with `vm/` in your vSphere config file. | +| Host | | The IP of the host system to schedule VMs in. If specified, the host system's pool will be used and the *Pool* parameter will be ignored. | + +{{% /tab %}} +{{% tab "Rancher prior to v2.3.3" %}} + +| Parameter | Required | Description | +|:------------------------|:--------:|:------------------------------------------------------------| +| Data Center | * | Name/path of the datacenter to create VMs in. | +| Pool | | Name/path of the resource pool to schedule the VMs in. If not specified, the default resource pool is used. | +| Host | | Name/path of the host system to schedule VMs in. If specified, the host system's pool will be used and the *Pool* parameter will be ignored. | +| Network | * | Name of the VM network to attach VMs to. | +| Data Store | * | Datastore to store the VM disks. | +| Folder | | Name of a folder in the datacenter to create the VMs in. Must already exist. The folder name should be prefaced with `vm/` in your vSphere config file. | +{{% /tab %}} +{{% /tabs %}} \ No newline at end of file diff --git a/content/rke/latest/en/config-options/cloud-providers/vsphere/_index.md b/content/rke/latest/en/config-options/cloud-providers/vsphere/_index.md index 2f37d14edb0..d73d2169d74 100644 --- a/content/rke/latest/en/config-options/cloud-providers/vsphere/_index.md +++ b/content/rke/latest/en/config-options/cloud-providers/vsphere/_index.md @@ -3,257 +3,27 @@ title: vSphere Cloud Provider weight: 254 --- +In order to provision Kubernetes clusters in vSphere with the RKE CLI, you must enable the vSphere cloud provider. + +The vSphere cloud provider must also be enabled in order to provision clusters with Rancher, which uses RKE as a library when provisioning [RKE clusters.]({{}}/rancher/v2.x/en/cluster-provisioning/rke-clusters/) + The [vSphere Cloud Provider](https://vmware.github.io/vsphere-storage-for-kubernetes/documentation/) interacts with VMware infrastructure (vCenter or standalone ESXi server) to provision and manage storage for persistent volumes in a Kubernetes cluster. -When provisioning Kubernetes using RKE CLI or using [RKE clusters]({{< baseurl >}}/rancher/v2.x/en/cluster-provisioning/rke-clusters/) in Rancher, the vSphere Cloud Provider can be enabled by configuring the `cloud_provider` directive in the cluster YAML file. - -### Prerequisites - -1. You'll need to have credentials of a vCenter/ESXi user account with privileges allowing the cloud provider to interact with the vSphere infrastructure to provision storage. Refer to [this document](https://vmware.github.io/vsphere-storage-for-kubernetes/documentation/vcp-roles.html) to create and assign a role with the required permissions in vCenter. -2. VMware Tools must be running in the Guest OS for all nodes in the cluster. -3. All nodes must be configured with disk UUIDs. This is required so that attached VMDKs present a consistent UUID to the VM, allowing the disk to be mounted properly. See [Enabling Disk UUIDs](#enabling-disk-uuids-for-vsphere-vms). - -## Clusters provisioned with RKE CLI - -To enable the vSphere Cloud Provider in the cluster, you must add the top-level `cloud_provider` directive to the cluster configuration file, set the `name` property to `vsphere` and add the `vsphereCloudProvider` directive containing the configuration matching your infrastructure. See the [configuration reference](#configuration-reference) for the gory details. - -## Clusters provisioned with Rancher - -When provisioning clusters in Rancher using the [vSphere node driver]({{< baseurl >}}/rancher/v2.x/en/cluster-provisioning/rke-clusters/node-pools/vsphere/) or on pre-created [custom nodes]({{< baseurl >}}/rancher/v2.x/en/cluster-provisioning/rke-clusters/custom-nodes/) the cluster YAML file must be modified in order to enable the cloud provider. - -1. Log in to the Rancher UI as admin user. -2. Navigate to **Clusters** in the **Global** view. -3. Click **Add Cluster** and select the **vSphere** infrastructure provider. -4. Assign a **Cluster Name**. -5. Assign **Member Roles** as required. -6. Expand **Cluster Options** and configure as required. -7. Set **Cloud Provider** option to `Custom`. - - {{< img "/img/rancher/vsphere-node-driver-cloudprovider.png" "vsphere-node-driver-cloudprovider">}} - -8. Click on **Edit as YAML** -9. Insert the following structure to the pre-populated cluster YAML. As of Rancher v2.3+, this structure must be placed under `rancher_kubernetes_engine_config`. In versions prior to v2.3, it has to be defined as a top level field. Note that the `name` *must* be set to `vsphere`. Refer to the [configuration reference](#configuration-reference) to learn about the properties of the `vsphereCloudProvider` directive. - - ```yaml - rancher_kubernetes_engine_config: # Required as of Rancher v2.3+ - cloud_provider: - name: vsphere - vsphereCloudProvider: - [Insert provider configuration] - ``` - -10. Configure the **Node Pools** per your requirements while ensuring to use a node template that enables disk UUIDs for the VMs (See [Annex - Enable disk UUIDs for vSphere VMs]). -11. Click on **Create** to start provisioning the VMs and Kubernetes services. - -## Configuration Reference - -> **Note:** This documentation reflects the new vSphere Cloud Provider configuration schema introduced in Kubernetes v1.9 which differs from previous versions. - -The vSphere configuration options are divided into 5 groups: - -* global -* virtual_center -* workspace -* disk -* network - -### global - -The main purpose of global options is to be able to define a common set of configuration parameters that will be inherited by all vCenters defined under the `virtual_center` directive unless explicitly defined there. - -Accordingly, the `global` directive accepts the same configuration options that are available under the `virtual_center` directive. Additionally it accepts a single parameter that can only be specified here: - -| global Options | Type | Required | Description | -|:---------------:|:-------:|:---------:|:-----------------------------------------------------------------------------:| -| insecure-flag | boolean | | Set to **true** if the vCenter/ESXi uses a self-signed certificate. | - -___ - -**Example:** - -```yaml -(...) - global: - insecure-flag: true -``` - -### virtual_center - -This configuration directive specifies the vCenters that are managing the nodes in the cluster. You must define at least one vCenter/ESXi server. If the nodes span multiple vCenters then all must be defined. - -Each vCenter is defined by adding a new entry under the `virtual_center` directive with the vCenter IP or FQDN as the name. All required parameters must be provided for each vCenter unless they are already defined under the `global` directive. - -| virtual_center Options | Type | Required | Description | -|:----------------------:|:--------:|:---------:|:-----------------------------------------------------------------------------:| -| user | string | * | vCenter/ESXi user used to authenticate with this server. | -| password | string | * | User's password. | -| port | string | | Port to use to connect to this server. Defaults to 443. | -| datacenters | string | * | Comma-separated list of all datacenters in which cluster nodes are running in.| -| soap-roundtrip-count | uint | | Round tripper count for API requests to the vCenter (num retries = value - 1).| - - -> The following additional options (introduced in Kubernetes v1.11) are not yet supported in RKE. - -| virtual_center Options | Type | Required | Description | -|:----------------------:|:--------:|:---------:|:-----------------------------------------------------------------------------:| -| secret-name | string | | Name of secret resource containing credential key/value pairs. Can be specified in lieu of user/password parameters.| -| secret-namespace | string | | Namespace in which the secret resource was created in. | -| ca-file | string | | Path to CA cert file used to verify the vCenter certificate. | - -___ - -**Example:** - -```yaml -(...) - virtual_center: - 172.158.111.1: {} # This vCenter inherits all it's properties from global options - 172.158.110.2: # All required options are set explicitly - user: vc-user - password: othersecret - datacenters: eu-west-2 -``` - -### workspace - -This configuration group specifies how storage for volumes is created in vSphere. -The following configuration options are available: - -| workspace Options | Type | Required | Description | -|:----------------------:|:--------:|:---------:|:-----------------------------------------------------------------------------:| -| server | string | * | IP or FQDN of the vCenter/ESXi that should be used for creating the volumes. Must match one of the vCenters defined under the `virtual_center` directive.| -| datacenter | string | * | Name of the datacenter that should be used for creating volumes. For ESXi enter *ha-datacenter*.| -| folder | string | * | Path of folder in which to create dummy VMs used for volume provisioning (relative from the root folder in vCenter), e.g. "kubernetes".| -| default-datastore | string | | Name of default datastore to place VMDKs if neither datastore or storage policy are specified in the volume options of a PVC. If datastore is located in a storage folder or is a member of a datastore cluster, specify the full path. | -| resourcepool-path | string | | Absolute or relative path to the resource pool where the dummy VMs for [Storage policy based provisioning](https://vmware.github.io/vsphere-storage-for-kubernetes/documentation/policy-based-mgmt.html) should be created. If a relative path is specified, it is resolved with respect to the datacenter's *host* folder. Examples: `//host//Resources/`, `Resources/`. For standalone ESXi specify `Resources`.| - -___ - -**Example:** - -```yaml -(...) - workspace: - server: 172.158.111.1 # matches IP of vCenter defined in the virtual_center block - datacenter: eu-west-1 - folder: kubernetes - default-datastore: ds-1 -``` - -### disk - -The following configuration options are available under the disk directive: - -| disk Options | Type | Required | Description | -|:--------------------:|:--------:|:---------:|:-----------------------------------------------------------------------------:| -| scsicontrollertype | string | | SCSI controller type to use when attaching block storage to VMs. Must be one of: *lsilogic-sas* or *pvscsi*. Default: *pvscsi*.| - -___ - -### network - -The following configuration options are available under the network directive: - -| network Options | Type | Required | Description | -|:-------------------:|:--------:|:---------:|:-----------------------------------------------------------------------------:| -| public-network | string | | Name of public **VM Network** to which the VMs in the cluster are connected. Used to determine public IP addresses of VMs.| - - -## Configuration Example - -Given the following: - -- VMs in the cluster are running in the same datacenter `eu-west-1` managed by the vCenter `vc.example.com`. -- The vCenter has a user `provisioner` with password `secret` with the required roles assigned, see [Prerequisites](#prerequisites). -- The vCenter has a datastore named `ds-1` which should be used to store the VMDKs for volumes. -- A `kubernetes` folder exists in vCenter. - -The corresponding configuration for the provider would then be as follows: - -```yaml -(...) -cloud_provider: - name: vsphere - vsphereCloudProvider: - virtual_center: - vc.example.com: - user: provisioner - password: secret - datacenters: eu-west-1 - workspace: - server: vc.example.com - folder: vm/kubernetes - default-datastore: ds-1 - datacenter: eu-west-1 - -``` - -## Annex - -### Enabling disk UUIDs for vSphere VMs - -Depending on whether you are provisioning the VMs using the [vSphere node driver]({{< baseurl >}}/rancher/v2.x/en/cluster-provisioning/rke-clusters/node-pools/vsphere) in Rancher or using your own scripts or third-party tools, there are different methods available to enable disk UUIDs for VMs. - -#### Using the Vsphere Console - -The required property can be set while creating or modifying VMs in the vSphere Console: - -1. For each VM navigate to the tab **VM Options** and click on **Edit Configuration**. -2. Add the parameter `disk.EnableUUID` with a value of **TRUE**. - - {{< img "/img/rke/vsphere-advanced-parameters.png" "vsphere-advanced-parameters">}} - -#### Using the GOVC CLI tool - -You can also modify properties of VMs with the [govc](https://github.com/vmware/govmomi/tree/master/govc) command-line tool to enable disk UUIDs: - -```sh -$ govc vm.change -vm -e disk.enableUUID=TRUE -``` - -#### Using Rancher node template - -When creating new clusters in Rancher using vSphere node templates, you can configure the template to automatically enable disk UUIDs for all VMs created for a cluster: - -1. Navigate to the **Node Templates** in the Rancher UI while logged in as admin user. - -2. Add or edit an existing vSphere node template. - -3. Under **Instance Options** click on **Add Parameter**. - -4. Enter `disk.enableUUID` as key with a value of **TRUE**. - - {{< img "/img/rke/vsphere-nodedriver-enable-uuid.png" "vsphere-nodedriver-enable-uuid">}} - -5. Click **Create** or **Save**. - -### Troubleshooting - -If you are experiencing issues while provisioning a cluster with enabled vSphere Cloud Provider or while creating vSphere volumes for your workloads, you should inspect the logs of the following K8s services: - -- controller-manager (Manages volumes in vCenter) -- kubelet: (Mounts vSphere volumes to pods) - -If your cluster is not configured with external [Cluster Logging]({{< baseurl >}}/rancher/v2.x/en/tools/logging/), you will need to SSH into nodes to get the logs of the `kube-controller-manager` (running on one of the control plane nodes) and the `kubelet` (pertaining to the node where the stateful pod has been scheduled). - -The easiest way to create a SSH session with a node is the Rancher CLI tool. - -1. [Configure the Rancher CLI]({{< baseurl >}}/rancher/v2.x/en/cli/) for your cluster. -2. Run the following command to get a shell to the corresponding nodes: - - ```sh -$ rancher ssh - ``` - -3. Inspect the logs of the controller-manager and kubelet containers looking for errors related to the vSphere cloud provider: - - ```sh - $ docker logs --since 15m kube-controller-manager - $ docker logs --since 15m kubelet - ``` - +This section describes how to enable the vSphere cloud provider. You will need to use the `cloud_provider` directive in the cluster YAML file. ### Related Links -- [vSphere Storage for Kubernetes](https://vmware.github.io/vsphere-storage-for-kubernetes/documentation/) -- [Kubernetes Persistent Volumes](https://kubernetes.io/docs/concepts/storage/persistent-volumes/) +- **Configuration:** For details on vSphere configuration in RKE, refer to the [configuration reference.]({{}}/rke/latest/en/config-options/cloud-providers/vsphere/config-reference) +- **Troubleshooting:** For guidance on troubleshooting a cluster with the vSphere cloud provider enabled, refer to the [troubleshooting section.]({{}}/rke/latest/en/config-options/cloud-providers/vsphere/troubleshooting) +- **Storage:** If you are setting up storage, see the [official vSphere documentation on storage for Kubernetes,](https://vmware.github.io/vsphere-storage-for-kubernetes/documentation/) or the [official Kubernetes documentation on persistent volumes.](https://kubernetes.io/docs/concepts/storage/persistent-volumes/) If you are using Rancher, refer to the [Rancher documentation on provisioning storage in vSphere.]({{}}/rancher/v2.x/en/cluster-admin/volumes-and-storage/examples/vsphere) +- **For Rancher users:** Refer to the Rancher documentation on [creating vSphere Kubernetes clusters]({{}}/rancher/v2.x/en/cluster-provisioning/rke-clusters/node-pools/vsphere) and [provisioning storage.]({{}}/rancher/v2.x/en/cluster-admin/volumes-and-storage/examples/vsphere) + +# Prerequisites + +- **Credentials:** You'll need to have credentials of a vCenter/ESXi user account with privileges allowing the cloud provider to interact with the vSphere infrastructure to provision storage. Refer to [this document](https://vmware.github.io/vsphere-storage-for-kubernetes/documentation/vcp-roles.html) to create and assign a role with the required permissions in vCenter. +- **VMware Tools** must be running in the Guest OS for all nodes in the cluster. +- **Disk UUIDs:** All nodes must be configured with disk UUIDs. This is required so that attached VMDKs present a consistent UUID to the VM, allowing the disk to be mounted properly. See the section on [enabling disk UUIDs.]({{}}/rke/latest/en/config-options/cloud-providers/vsphere/enabling-uuid) + +# Enabling the vSphere Provider with the RKE CLI + +To enable the vSphere Cloud Provider in the cluster, you must add the top-level `cloud_provider` directive to the cluster configuration file, set the `name` property to `vsphere` and add the `vsphereCloudProvider` directive containing the configuration matching your infrastructure. See the [configuration reference]({{}}/rke/latest/en/config-options/cloud-providers/vsphere/config-reference) for the gory details. \ No newline at end of file diff --git a/content/rke/latest/en/config-options/cloud-providers/vsphere/config-reference/_index.md b/content/rke/latest/en/config-options/cloud-providers/vsphere/config-reference/_index.md new file mode 100644 index 00000000000..c0da0176c69 --- /dev/null +++ b/content/rke/latest/en/config-options/cloud-providers/vsphere/config-reference/_index.md @@ -0,0 +1,144 @@ +--- +title: vSphere Configuration Reference +weight: 3 +--- + +This section shows an example of how to configure the vSphere cloud provider. + +The vSphere cloud provider must be enabled to allow dynamic provisioning of volumes. + +For more details on deploying a Kubernetes cluster on vSphere, refer to the [official cloud provider documentation.](https://cloud-provider-vsphere.sigs.k8s.io/tutorials/kubernetes-on-vsphere-with-kubeadm.html) + +> **Note:** This documentation reflects the new vSphere Cloud Provider configuration schema introduced in Kubernetes v1.9 which differs from previous versions. + +# vSphere Configuration Example + +Given the following: + +- VMs in the cluster are running in the same datacenter `eu-west-1` managed by the vCenter `vc.example.com`. +- The vCenter has a user `provisioner` with password `secret` with the required roles assigned, see [Prerequisites](#prerequisites). +- The vCenter has a datastore named `ds-1` which should be used to store the VMDKs for volumes. +- A `vm/kubernetes` folder exists in vCenter. + +The corresponding configuration for the provider would then be as follows: + +```yaml +(...) +cloud_provider: + name: vsphere + vsphereCloudProvider: + virtual_center: + vc.example.com: + user: provisioner + password: secret + port: 443 + datacenters: /us-west-1 + workspace: + server: vc.example.com + folder: /us-west-1/folder/myvmfolder + default-datastore: /us-west-1/datastore/ds-1 + datacenter: /us-west-1 + resourcepool-path: /us-west-1/host/hn1/resources/myresourcepool + +``` +# Configuration Options + +The vSphere configuration options are divided into 5 groups: + +* [global](#global) +* [virtual_center](#virtual_center) +* [workspace](#workspace) +* [disk](#disk) +* [network](#network) + +### global + +The main purpose of global options is to be able to define a common set of configuration parameters that will be inherited by all vCenters defined under the `virtual_center` directive unless explicitly defined there. + +Accordingly, the `global` directive accepts the same configuration options that are available under the `virtual_center` directive. Additionally it accepts a single parameter that can only be specified here: + +| global Options | Type | Required | Description | +|:---------------:|:-------:|:---------:|:---------| +| insecure-flag | boolean | | Set to **true** if the vCenter/ESXi uses a self-signed certificate. | + +**Example:** + +```yaml +(...) + global: + insecure-flag: true +``` + +### virtual_center + +This configuration directive specifies the vCenters that are managing the nodes in the cluster. You must define at least one vCenter/ESXi server. If the nodes span multiple vCenters then all must be defined. + +Each vCenter is defined by adding a new entry under the `virtual_center` directive with the vCenter IP or FQDN as the name. All required parameters must be provided for each vCenter unless they are already defined under the `global` directive. + +| virtual_center Options | Type | Required | Description | +|:----------------------:|:--------:|:---------:|:-----------| +| user | string | * | vCenter/ESXi user used to authenticate with this server. | +| password | string | * | User's password. | +| port | string | | Port to use to connect to this server. Defaults to 443. | +| datacenters | string | * | Comma-separated list of all datacenters in which cluster nodes are running in. | +| soap-roundtrip-count | uint | | Round tripper count for API requests to the vCenter (num retries = value - 1). | + +> The following additional options (introduced in Kubernetes v1.11) are not yet supported in RKE. + +| virtual_center Options | Type | Required | Description | +|:----------------------:|:--------:|:---------:|:-------| +| secret-name | string | | Name of secret resource containing credential key/value pairs. Can be specified in lieu of user/password parameters.| +| secret-namespace | string | | Namespace in which the secret resource was created in. | +| ca-file | string | | Path to CA cert file used to verify the vCenter certificate. | + +**Example:** + +```yaml +(...) + virtual_center: + 172.158.111.1: {} # This vCenter inherits all it's properties from global options + 172.158.110.2: # All required options are set explicitly + user: vc-user + password: othersecret + datacenters: eu-west-2 +``` + +### workspace + +This configuration group specifies how storage for volumes is created in vSphere. +The following configuration options are available: + +| workspace Options | Type | Required | Description | +|:----------------------:|:--------:|:---------:|:---------| +| server | string | * | IP or FQDN of the vCenter/ESXi that should be used for creating the volumes. Must match one of the vCenters defined under the `virtual_center` directive.| +| datacenter | string | * | Name of the datacenter that should be used for creating volumes. For ESXi enter *ha-datacenter*.| +| folder | string | * | Path of folder in which to create dummy VMs used for volume provisioning (relative from the root folder in vCenter), e.g. "vm/kubernetes".| +| default-datastore | string | | Name of default datastore to place VMDKs if neither datastore or storage policy are specified in the volume options of a PVC. If datastore is located in a storage folder or is a member of a datastore cluster, specify the full path. | +| resourcepool-path | string | | Absolute or relative path to the resource pool where the dummy VMs for [Storage policy based provisioning](https://vmware.github.io/vsphere-storage-for-kubernetes/documentation/policy-based-mgmt.html) should be created. If a relative path is specified, it is resolved with respect to the datacenter's *host* folder. Examples: `//host//Resources/`, `Resources/`. For standalone ESXi specify `Resources`. | + +**Example:** + +```yaml +(...) + workspace: + server: 172.158.111.1 # matches IP of vCenter defined in the virtual_center block + datacenter: eu-west-1 + folder: vm/kubernetes + default-datastore: ds-1 +``` + +### disk + +The following configuration options are available under the disk directive: + +| disk Options | Type | Required | Description | +|:--------------------:|:--------:|:---------:|:----------------| +| scsicontrollertype | string | | SCSI controller type to use when attaching block storage to VMs. Must be one of: *lsilogic-sas* or *pvscsi*. Default: *pvscsi*. | + +### network + +The following configuration options are available under the network directive: + +| network Options | Type | Required | Description | +|:-------------------:|:--------:|:---------:|:-----------------------------------------------------------------------------| +| public-network | string | | Name of public **VM Network** to which the VMs in the cluster are connected. Used to determine public IP addresses of VMs.| \ No newline at end of file diff --git a/content/rke/latest/en/config-options/cloud-providers/vsphere/enabling-uuid/_index.md b/content/rke/latest/en/config-options/cloud-providers/vsphere/enabling-uuid/_index.md new file mode 100644 index 00000000000..0277e725a07 --- /dev/null +++ b/content/rke/latest/en/config-options/cloud-providers/vsphere/enabling-uuid/_index.md @@ -0,0 +1,35 @@ +--- +title: Enabling Disk UUIDs for vSphere VMs +weight: 2 +--- + +In order to provision nodes with RKE, all nodes must be configured with disk UUIDs. This is required so that attached VMDKs present a consistent UUID to the VM, allowing the disk to be mounted properly. + +Depending on whether you are provisioning the VMs using the [vSphere node driver]({{}}/rancher/v2.x/en/cluster-provisioning/rke-clusters/node-pools/vsphere) in Rancher or using your own scripts or third-party tools, there are different methods available to enable disk UUIDs for VMs: + +- [Using the vSphere console](#using-the-vsphere-console) +- [Using the GOVC CLI tool](#using-the-govc-cli-tool) +- [Using a Rancher node template](#using-a-rancher-node-template) + +### Using the vSphere Console + +The required property can be set while creating or modifying VMs in the vSphere Console: + +1. For each VM navigate to the tab **VM Options** and click on **Edit Configuration**. +2. Add the parameter `disk.EnableUUID` with a value of **TRUE**. + + ![vsphere-advanced-parameters]({{< baseurl >}}/img/rke/vsphere-advanced-parameters.png) + +### Using the GOVC CLI tool + +You can also modify properties of VMs with the [govc](https://github.com/vmware/govmomi/tree/master/govc) command-line tool to enable disk UUIDs: + +```sh +$ govc vm.change -vm -e disk.enableUUID=TRUE +``` + +### Using a Rancher Node Template + +In Rancher v2.0.4+, disk UUIDs are enabled in vSphere node templates by default. + +If you are using Rancher prior to v2.0.4, refer to the [Rancher documentation.]({{}}/rancher/v2.x/en/cluster-provisioning/rke-clusters/node-pools/vsphere/#enabling-disk-uuids-with-a-node-template) for details on how to enable a UUID with a Rancher node template. \ No newline at end of file diff --git a/content/rke/latest/en/config-options/cloud-providers/vsphere/troubleshooting/_index.md b/content/rke/latest/en/config-options/cloud-providers/vsphere/troubleshooting/_index.md new file mode 100644 index 00000000000..6d2cffca67f --- /dev/null +++ b/content/rke/latest/en/config-options/cloud-providers/vsphere/troubleshooting/_index.md @@ -0,0 +1,27 @@ +--- +title: Troubleshooting vSphere Clusters +weight: 4 +--- + +If you are experiencing issues while provisioning a cluster with enabled vSphere Cloud Provider or while creating vSphere volumes for your workloads, you should inspect the logs of the following K8s services: + +- controller-manager (Manages volumes in vCenter) +- kubelet: (Mounts vSphere volumes to pods) + +If your cluster is not configured with external [Cluster Logging]({{< baseurl >}}/rancher/v2.x/en/tools/logging/), you will need to SSH into nodes to get the logs of the `kube-controller-manager` (running on one of the control plane nodes) and the `kubelet` (pertaining to the node where the stateful pod has been scheduled). + +The easiest way to create a SSH session with a node is the Rancher CLI tool. + +1. [Configure the Rancher CLI]({{< baseurl >}}/rancher/v2.x/en/cli/) for your cluster. +2. Run the following command to get a shell to the corresponding nodes: + + ```sh +$ rancher ssh + ``` + +3. Inspect the logs of the controller-manager and kubelet containers looking for errors related to the vSphere cloud provider: + + ```sh + $ docker logs --since 15m kube-controller-manager + $ docker logs --since 15m kubelet + ``` diff --git a/content/rke/latest/en/config-options/secrets-encryption/_index.md b/content/rke/latest/en/config-options/secrets-encryption/_index.md new file mode 100644 index 00000000000..9d447592c0a --- /dev/null +++ b/content/rke/latest/en/config-options/secrets-encryption/_index.md @@ -0,0 +1,174 @@ +--- +title: Encrypting Secret Data at Rest +weight: 230 +--- + +As of version `v0.3.1` RKE adds the support for managing secret data encryption at rest, which is [supported by Kubernetes](https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/#before-you-begin) since version `v1.13`. + +At-rest data encryption is required for: + +- Compliance requirements +- Additional layer of security +- Reduce security impact of etcd node compromise +- Reduce security impact of etcd backups compromise +- Ability to use external Key Management Systems + +RKE provides users with two paths of configuration to enable at-rest data encryption: + +- Managed at-rest data encryption +- Custom configuration for at-rest data encryption + +Both configuration options can be added during initial cluster provisioning or by updating an exsiting cluster. + +To utilize this feature, a new field `secrets_encryption_config` is added to the [Kubernetes API service configuration]({{}}//rke/latest/en/config-options/services/#kubernetes-api-server). A full custom configuration looks like this: + +```yaml +services: + kube-api: + secrets_encryption_config: + enabled: true + custom_config: + apiVersion: apiserver.config.k8s.io/v1 + kind: EncryptionConfiguration + resources: + - resources: + - secrets + providers: + - aescbc: + keys: + - name: k-fw5hn + secret: RTczRjFDODMwQzAyMDVBREU4NDJBMUZFNDhCNzM5N0I= + - identity: {} + +``` +# Managed At-Rest Data Encryption + +Enabling and disabling at-rest data encryption in Kubernetes is a relatively complex process that requires several steps to be performed by the Kubernetes cluster administrator. The managed configuration aims to reduce this overhead and provides a simple abstraction layer to manage the process. + +### Enable Encryption +Managed at-rest data encryption is disabled by default and can be enabled by using the following configuration: + +```yaml +services: + kube-api: + secrets_encryption_config: + enabled: true +``` +Once enabled, RKE will perform the following [actions](https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/#encrypting-your-data) to enable at-rest data encryption: + +- Generate a new random 32-bit encryption key +- Generate an encryption provider configuration file using the new key The default [provider](https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/#providers) used is `aescbc` +- Deploy the provider configuration file to all nodes with `controlplane` role +- Update the `kube-apiserver` container arguments to point to the provider configuration file. +- Restart the `kube-apiserver` container. + +After the `kube-api server` is restarted, data encryption is enabled. However, all existing secrets are still stored in plain text. RKE will [rewrite](https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/#ensure-all-secrets-are-encrypted) all secrets to ensure encryption is fully in effect. + +### Disable Encryption +To disable encryption, you can either set the `enabled` flag to `false`, or simply remove the `secrets_encryption_config` block entirely from cluster.yml. + +```yaml +services: + kube-api: + secrets_encryption_config: + enabled: false +``` + +Once encryption is disabled in `cluster.yml`, RKE will perform the following [actions](https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/#encrypting-your-data) to disable encryption in your cluster: + +- Generate a new provider configuration file with the no-encryption `identity{}` provider as the first provider, and the previous `aescbc` set in the second place. This will allow Kubernetes to use the first entry to write the secrets, and the second one to decrypt them. +- Deploy the new provider configuration and restart `kube-apiserver`. +- Rewrite all secrets. This is required because, at this point, new data will be written to disk in plain text, but the existing data is still encrypted using the old provider. By rewriting all secrets, RKE ensures that all stored data is decrypted. +- Update `kube-apiserver` arguments to remove the encryption provider configuration and restart the `kube-apiserver`. +- Remove the provider configuration file. + + +# Key Rotation +Sometimes there is a need to rotate encryption config in your cluster. For example, the key is compromised. There are two ways to rotate the keys: with an RKE CLI command, or by disabling and re-enabling encryption in `cluster.yml`. + +### Rotating Keys with the RKE CLI + +With managed configuration, RKE CLI has the ability to perform the key rotation process documented [here](https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/#rotating-a-decryption-key) with one command. To perform this operation, the following subcommand is used: +```bash +$ ./rke encrypt rotate-key --help +NAME: + rke encrypt rotate-key - Rotate cluster encryption provider key + +USAGE: + rke encrypt rotate-key [command options] [arguments...] + +OPTIONS: + --config value Specify an alternate cluster YAML file (default: "cluster.yml") [$RKE_CONFIG] + --ssh-agent-auth Use SSH Agent Auth defined by SSH_AUTH_SOCK + --ignore-docker-version Disable Docker version check + +``` +This command will perform the following actions: + +- Generate a new random 32-bit encryption key +- Generate a new provider configuration with the new key as the first provider and the second key as the second provider. When the secrets are rewritten, the first key will be used to encrypt the data on the write operation, while the second key (the old key) will be used to decrypt the stored data during the the read operation +- Deploy the new provider configuration to all `controlplane` nodes and restart the `kube-apiserver` +- Rewrite all secrets. This process will re-encrypt all the secrets with the new key. +- Update the configuration to remove the old key and restart the `kube-apiserver` + +### Rotating Keys by Disabling and Re-enabling Encryption in cluster.yml + +For a cluster with encryption enabled, you can rotate the encryption keys by updating `cluster.yml`. If you enable and re-enable the data encryption in the `cluster.yml`, RKE will not reuse old keys. Instead, it will generate new keys every time, yielding the same result as a key rotation with the RKE CLI. + +# Custom At-Rest Data Encryption Configuration +With managed configuration, RKE provides the user with a very simple way to enable and disable encryption with minimal interaction and configuration. However, it doesn't allow for any customization to the configuration. + +With custom encryption configuration, RKE allows the user to provide their own configuration. Although RKE will help the user to deploy the configuration and rewrite the secrets if needed, it doesn't provide a configuration validation on user's behalf. It's the user responsibility to make sure their configuration is valid. + +>**Warning:** Using invalid Encryption Provider Configuration could cause several issues with your cluster, ranging from crashing the Kubernetes API service, `kube-api`, to completely losing access to encrypted data. + +### Example: Using Custom Encryption Configuration with Amazon KMS + +An example for custom configuration would be enabling an external key management system like [Amazon KMS](https://aws.amazon.com/kms/). The following is an example of the configuration for AWS KMS: + +```yaml + +services: + kube-api: + extra_binds: + - "/var/run/kmsplugin/:/var/run/kmsplugin/" + secrets_encryption_config: + enabled: true + custom_config: + apiVersion: apiserver.config.k8s.io/v1 + kind: EncryptionConfiguration + resources: + - resources: + - secrets + providers: + - kms: + name: aws-encryption-provider + endpoint: unix:///var/run/kmsplugin/socket.sock + cachesize: 1000 + timeout: 3s + - identity: {} +``` + +Documentation for AWS KMS can be found [here](https://github.com/kubernetes-sigs/aws-encryption-provider). When Custom Configuration is set to to enable the AWS KMS provider, you should consider the following points: + +- Since RKE runs the `kube-api` service in a container, it's required that you use the `extra_binds` feature to bind-mount the KMS provider socket location inside the `kube-api` container. +- The AWS KMS provider runs as a pod in the cluster. Therefor, the proper way to enable it is to: + 1. Deploy your cluster with at-rest encryption disabled. + 2. Deploy the KMS pod and make sure it's working correctly. + 3. Update your cluster with the custom encryption configuration to utilize the KMS provider. +- Kube API connects to the KMS provider using a Unix socket. You should configure your KMS deployment to run pods on all `controlplane` nodes in the cluster. +- Your `controlplane` node should be configured with an AMI profile that has access to the KMS key you used in your configuration. + +### How to Prevent Restore Failures after Rotating Keys +It's important to understand that enabling encryption for you cluster means that you can no longer access encrypted data in your etcd database and/or etcd database backups without using your encryption keys. + +The encryption configuration is stored in the cluster state file `cluster.rkestate`, which is decoupled from the etcd backups. For example, in any of the following backup cases, the restore process will fail: + +- The snapshot is taken while encryption is enabled and restored when it's disabled. In this case, the encryption keys are no longer stored in the cluster state. +- The snapshot is taken before the keys are rotated and restore is attempted after. In this case, the old keys used for encryption at the time of the snapshot no longer exist in the cluster state file. + +Therefore, we recommend that when you enable or disable encryption, or when you rotate keys, you should [create a snapshot]({{}}/rke/latest/en/etcd-snapshots/one-time-snapshots/) so that your backup requires the same keys that you have access to. + +This also means you should not rotate the keys during the restore process, because you would lose the encryption keys in `cluster.rkestate`. + +The same applies to the custom configuration use case, however in this case it will depend on the user-provided encryption configuration. diff --git a/content/rke/latest/en/config-options/services/_index.md b/content/rke/latest/en/config-options/services/_index.md index 8446aeb52b4..17af63348bf 100644 --- a/content/rke/latest/en/config-options/services/_index.md +++ b/content/rke/latest/en/config-options/services/_index.md @@ -22,7 +22,7 @@ Kubernetes uses [etcd](https://etcd.io/) as a store for cluster state and data. RKE supports running etcd in a single node mode or in HA cluster mode. It also supports adding and removing etcd nodes to the cluster. -You can enable etcd to [take recurring snapshots]({{< baseurl >}}/rke/latest/en/etcd-snapshots/#recurring-snapshots). These snapshots can be used to [restore etcd]({{< baseurl >}}/rke/latest/en/etcd-snapshots/#etcd-disaster-recovery). +You can enable etcd to [take recurring snapshots]({{< baseurl >}}/rke/latest/en/etcd-snapshots/#recurring-snapshots). These snapshots can be used to [restore etcd]({{< baseurl >}}/rke/latest/en/etcd-snapshots/#etcd-disaster-recovery). By default, RKE will deploy a new etcd service, but you can also run Kubernetes with an [external etcd service]({{< baseurl >}}/rke/latest/en/config-options/services/external-etcd/). @@ -44,6 +44,8 @@ services: # Enable AlwaysPullImages Admission controller plugin # Available as of v0.2.0 always_pull_images: false + secrets_encryption_config: + enabled: true ``` ### Kubernetes API Server Options @@ -55,7 +57,7 @@ RKE supports the following options for the `kube-api` service : - **Pod Security Policy** (`pod_security_policy`) - An option to enable the [Kubernetes Pod Security Policy](https://kubernetes.io/docs/concepts/policy/pod-security-policy/). By default, we do not enable pod security policies as it is set to `false`. > **Note:** If you set `pod_security_policy` value to `true`, RKE will configure an open policy to allow any pods to work on the cluster. You will need to configure your own policies to fully utilize PSP. - **Always Pull Images** (`always_pull_images`) - Enable `AlwaysPullImages` Admission controller plugin. Enabling `AlwaysPullImages` is a security best practice. It forces Kubernetes to validate the image and pull credentials with the remote image registry. Local image layer cache will still be used, but it does add a small bit of overhead when launching containers to pull and compare image hashes. _Note: Available as of v0.2.0_ - +- **Secrets Encryption Config** (`secrets_encryption_config`) - Manage Kubernetes at-rest data encryption. Documented [here]({{< baseurl >}}//rke/latest/en/config-options/secrets-encryption) ## Kubernetes Controller Manager > **Note for Rancher 2 users** If you are configuring Cluster Options using a [Config File]({{< baseurl >}}/rancher/v2.x/en/cluster-provisioning/rke-clusters/options/#config-file) when creating [Rancher Launched Kubernetes]({{< baseurl >}}/rancher/v2.x/en/cluster-provisioning/rke-clusters/), the names of services should contain underscores only: `kube_controller`. This only applies to Rancher v2.0.5 and v2.0.6. @@ -92,6 +94,8 @@ services: cluster_dns_server: 10.43.0.10 # Fail if swap is on fail_swap_on: false + # Generate per node serving certificate + generate_serving_certificate: false ``` ### Kubelet Options @@ -101,6 +105,15 @@ RKE supports the following options for the `kubelet` service: - **Cluster Domain** (`cluster_domain`) - The [base domain](https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/) for the cluster. All services and DNS records created on the cluster. By default, the domain is set to `cluster.local`. - **Cluster DNS Server** (`cluster_dns_server`) - The IP address assigned to the DNS service endpoint within the cluster. DNS queries will be sent to this IP address which is used by KubeDNS. The default value for this option is `10.43.0.10` - **Fail if Swap is On** (`fail_swap_on`) - In Kubernetes, the default behavior for the kubelet is to **fail** if swap is enabled on the node. RKE does **not** follow this default and allows deployments on nodes with swap enabled. By default, the value is `false`. If you'd like to revert to the default kubelet behavior, set this option to `true`. +- **Generate Serving Certificate** (`generate_serving_certificate`) - Generate a certificate signed by the `kube-ca` Certificate Authority for the kubelet to use as a server certificate. The default value for this option is `false`. Before enabling this option, please read [the requirements](#kubelet-serving-certificate-requirements) + +### Kubelet Serving Certificate Requirements + +If `hostname_override` is configured for one or more nodes in `cluster.yml`, please make sure the correct IP address is configured in `address` (and the internal address in `internal_address`) to make sure the generated certificate contains the correct IP address(es). + +An example of an error situation is an EC2 instance where the the public IP address is configured in `address`, and `hostname_override` is used, the connection between `kube-apiserver` and `kubelet` will fail because the `kubelet` will be contacted on the private IP address and the generated certificate will not be valid (the error `x509: certificate is valid for value_in_address, not private_ip` will be seen). The resolution is to provide the internal IP address in `internal_address`. + +For more information on host overrides, refer to the [node configuration page.]({{}}/rke/latest/en/config-options/nodes/#overriding-the-hostname) ## Kubernetes Scheduler