Revise and update vSphere docs for Rancher v2.3.3 changes (#2016)

* Revise and update vSphere docs for Rancher v2.3.3 changes

* Edit vSphere docs

* Update _index.md

* Update _index.md

* Update _index.md
This commit is contained in:
Catherine Luse
2019-11-27 08:06:59 -07:00
committed by Denise
parent 8810b6d056
commit 08b6cceabc
10 changed files with 708 additions and 397 deletions
@@ -3,256 +3,27 @@ title: vSphere Cloud Provider
weight: 254
---
In order to provision Kubernetes clusters in vSphere with the RKE CLI, you must enable the vSphere cloud provider.
The vSphere cloud provider must also be enabled in order to provision clusters with Rancher, which uses RKE as a library when provisioning [RKE clusters.]({{<baseurl>}}/rancher/v2.x/en/cluster-provisioning/rke-clusters/)
The [vSphere Cloud Provider](https://vmware.github.io/vsphere-storage-for-kubernetes/documentation/) interacts with VMware infrastructure (vCenter or standalone ESXi server) to provision and manage storage for persistent volumes in a Kubernetes cluster.
When provisioning Kubernetes using RKE CLI or using [RKE clusters]({{< baseurl >}}/rancher/v2.x/en/cluster-provisioning/rke-clusters/) in Rancher, the vSphere Cloud Provider can be enabled by configuring the `cloud_provider` directive in the cluster YAML file.
### Prerequisites
1. You'll need to have credentials of a vCenter/ESXi user account with privileges allowing the cloud provider to interact with the vSphere infrastructure to provision storage. Refer to [this document](https://vmware.github.io/vsphere-storage-for-kubernetes/documentation/vcp-roles.html) to create and assign a role with the required permissions in vCenter.
2. VMware Tools must be running in the Guest OS for all nodes in the cluster.
3. All nodes must be configured with disk UUIDs. This is required so that attached VMDKs present a consistent UUID to the VM, allowing the disk to be mounted properly. See [Enabling Disk UUIDs](#enabling-disk-uuids-for-vsphere-vms).
## Clusters provisioned with RKE CLI
To enable the vSphere Cloud Provider in the cluster, you must add the top-level `cloud_provider` directive to the cluster configuration file, set the `name` property to `vsphere` and add the `vsphereCloudProvider` directive containing the configuration matching your infrastructure. See the [configuration reference](#configuration-reference) for the gory details.
## Clusters provisioned with Rancher
When provisioning clusters in Rancher using the [vSphere node driver]({{< baseurl >}}/rancher/v2.x/en/cluster-provisioning/rke-clusters/node-pools/vsphere/) or on pre-created [custom nodes]({{< baseurl >}}/rancher/v2.x/en/cluster-provisioning/rke-clusters/custom-nodes/) the cluster YAML file must be modified in order to enable the cloud provider.
1. Log in to the Rancher UI as admin user.
2. Navigate to **Clusters** in the **Global** view.
3. Click **Add Cluster** and select the **vSphere** infrastructure provider.
4. Assign a **Cluster Name**.
5. Assign **Member Roles** as required.
6. Expand **Cluster Options** and configure as required.
7. Set **Cloud Provider** option to `Custom`.
![vsphere-node-driver-cloudprovider]({{< baseurl >}}/img/rancher/vsphere-node-driver-cloudprovider.png)
8. Click on **Edit as YAML**
9. Insert the following top-level structure to the pre-populated cluster YAML. Note that the `name` *must* be set to `vsphere`. Refer to the [configuration reference](#configuration-reference) to learn about the properties of the `vsphereCloudProvider` directive.
```yaml
cloud_provider:
name: vsphere
vsphereCloudProvider:
[Insert provider configuration]
```
10. Configure the **Node Pools** per your requirements while ensuring to use a node template that enables disk UUIDs for the VMs (See [Annex - Enable disk UUIDs for vSphere VMs]).
11. Click on **Create** to start provisioning the VMs and Kubernetes services.
## Configuration Reference
> **Note:** This documentation reflects the new vSphere Cloud Provider configuration schema introduced in Kubernetes v1.9 which differs from previous versions.
The vSphere configuration options are divided into 5 groups:
* global
* virtual_center
* workspace
* disk
* network
### global
The main purpose of global options is to be able to define a common set of configuration parameters that will be inherited by all vCenters defined under the `virtual_center` directive unless explicitly defined there.
Accordingly, the `global` directive accepts the same configuration options that are available under the `virtual_center` directive. Additionally it accepts a single parameter that can only be specified here:
| global Options | Type | Required | Description |
|:---------------:|:-------:|:---------:|:-----------------------------------------------------------------------------:|
| insecure-flag | boolean | | Set to **true** if the vCenter/ESXi uses a self-signed certificate. |
___
**Example:**
```yaml
(...)
global:
insecure-flag: true
```
### virtual_center
This configuration directive specifies the vCenters that are managing the nodes in the cluster. You must define at least one vCenter/ESXi server. If the nodes span multiple vCenters then all must be defined.
Each vCenter is defined by adding a new entry under the `virtual_center` directive with the vCenter IP or FQDN as the name. All required parameters must be provided for each vCenter unless they are already defined under the `global` directive.
| virtual_center Options | Type | Required | Description |
|:----------------------:|:--------:|:---------:|:-----------------------------------------------------------------------------:|
| user | string | * | vCenter/ESXi user used to authenticate with this server. |
| password | string | * | User's password. |
| port | string | | Port to use to connect to this server. Defaults to 443. |
| datacenters | string | * | Comma-separated list of all datacenters in which cluster nodes are running in.|
| soap-roundtrip-count | uint | | Round tripper count for API requests to the vCenter (num retries = value - 1).|
> The following additional options (introduced in Kubernetes v1.11) are not yet supported in RKE.
| virtual_center Options | Type | Required | Description |
|:----------------------:|:--------:|:---------:|:-----------------------------------------------------------------------------:|
| secret-name | string | | Name of secret resource containing credential key/value pairs. Can be specified in lieu of user/password parameters.|
| secret-namespace | string | | Namespace in which the secret resource was created in. |
| ca-file | string | | Path to CA cert file used to verify the vCenter certificate. |
___
**Example:**
```yaml
(...)
virtual_center:
172.158.111.1: {} # This vCenter inherits all it's properties from global options
172.158.110.2: # All required options are set explicitly
user: vc-user
password: othersecret
datacenters: eu-west-2
```
### workspace
This configuration group specifies how storage for volumes is created in vSphere.
The following configuration options are available:
| workspace Options | Type | Required | Description |
|:----------------------:|:--------:|:---------:|:-----------------------------------------------------------------------------:|
| server | string | * | IP or FQDN of the vCenter/ESXi that should be used for creating the volumes. Must match one of the vCenters defined under the `virtual_center` directive.|
| datacenter | string | * | Name of the datacenter that should be used for creating volumes. For ESXi enter *ha-datacenter*.|
| folder | string | * | Path of folder in which to create dummy VMs used for volume provisioning (relative from the root folder in vCenter), e.g. "kubernetes".|
| default-datastore | string | | Name of default datastore to place VMDKs if neither datastore or storage policy are specified in the volume options of a PVC. If datastore is located in a storage folder or is a member of a datastore cluster, specify the full path. |
| resourcepool-path | string | | Absolute or relative path to the resource pool where the dummy VMs for [Storage policy based provisioning](https://vmware.github.io/vsphere-storage-for-kubernetes/documentation/policy-based-mgmt.html) should be created. If a relative path is specified, it is resolved with respect to the datacenter's *host* folder. Examples: `/<dataCenter>/host/<hostOrClusterName>/Resources/<poolName>`, `Resources/<poolName>`. For standalone ESXi specify `Resources`.|
___
**Example:**
```yaml
(...)
workspace:
server: 172.158.111.1 # matches IP of vCenter defined in the virtual_center block
datacenter: eu-west-1
folder: kubernetes
default-datastore: ds-1
```
### disk
The following configuration options are available under the disk directive:
| disk Options | Type | Required | Description |
|:--------------------:|:--------:|:---------:|:-----------------------------------------------------------------------------:|
| scsicontrollertype | string | | SCSI controller type to use when attaching block storage to VMs. Must be one of: *lsilogic-sas* or *pvscsi*. Default: *pvscsi*.|
___
### network
The following configuration options are available under the network directive:
| network Options | Type | Required | Description |
|:-------------------:|:--------:|:---------:|:-----------------------------------------------------------------------------:|
| public-network | string | | Name of public **VM Network** to which the VMs in the cluster are connected. Used to determine public IP addresses of VMs.|
## Configuration Example
Given the following:
- VMs in the cluster are running in the same datacenter `eu-west-1` managed by the vCenter `vc.example.com`.
- The vCenter has a user `provisioner` with password `secret` with the required roles assigned, see [Prerequisites](#prerequisites).
- The vCenter has a datastore named `ds-1` which should be used to store the VMDKs for volumes.
- A `kubernetes` folder exists in vCenter.
The corresponding configuration for the provider would then be as follows:
```yaml
(...)
cloud_provider:
name: vsphere
vsphereCloudProvider:
virtual_center:
vc.example.com:
user: provisioner
password: secret
datacenters: eu-west-1
workspace:
server: vc.example.com
folder: kubernetes
default-datastore: ds-1
datacenter: eu-west-1
```
## Annex
### Enabling disk UUIDs for vSphere VMs
Depending on whether you are provisioning the VMs using the [vSphere node driver]({{< baseurl >}}/rancher/v2.x/en/cluster-provisioning/rke-clusters/node-pools/vsphere) in Rancher or using your own scripts or third-party tools, there are different methods available to enable disk UUIDs for VMs.
#### Using the Vsphere Console
The required property can be set while creating or modifying VMs in the vSphere Console:
1. For each VM navigate to the tab **VM Options** and click on **Edit Configuration**.
2. Add the parameter `disk.EnableUUID` with a value of **TRUE**.
![vsphere-advanced-parameters]({{< baseurl >}}/img/rke/vsphere-advanced-parameters.png)
#### Using the GOVC CLI tool
You can also modify properties of VMs with the [govc](https://github.com/vmware/govmomi/tree/master/govc) command-line tool to enable disk UUIDs:
```sh
$ govc vm.change -vm <vm-path> -e disk.enableUUID=TRUE
```
#### Using Rancher node template
When creating new clusters in Rancher using vSphere node templates, you can configure the template to automatically enable disk UUIDs for all VMs created for a cluster:
1. Navigate to the **Node Templates** in the Rancher UI while logged in as admin user.
2. Add or edit an existing vSphere node template.
3. Under **Instance Options** click on **Add Parameter**.
4. Enter `disk.enableUUID` as key with a value of **TRUE**.
![vsphere-nodedriver-enable-uuid]({{< baseurl >}}/img/rke/vsphere-nodedriver-enable-uuid.png)
5. Click **Create** or **Save**.
### Troubleshooting
If you are experiencing issues while provisioning a cluster with enabled vSphere Cloud Provider or while creating vSphere volumes for your workloads, you should inspect the logs of the following K8s services:
- controller-manager (Manages volumes in vCenter)
- kubelet: (Mounts vSphere volumes to pods)
If your cluster is not configured with external [Cluster Logging]({{< baseurl >}}/rancher/v2.x/en/tools/logging/), you will need to SSH into nodes to get the logs of the `kube-controller-manager` (running on one of the control plane nodes) and the `kubelet` (pertaining to the node where the stateful pod has been scheduled).
The easiest way to create a SSH session with a node is the Rancher CLI tool.
1. [Configure the Rancher CLI]({{< baseurl >}}/rancher/v2.x/en/cli/) for your cluster.
2. Run the following command to get a shell to the corresponding nodes:
```sh
$ rancher ssh <nodeName>
```
3. Inspect the logs of the controller-manager and kubelet containers looking for errors related to the vSphere cloud provider:
```sh
$ docker logs --since 15m kube-controller-manager
$ docker logs --since 15m kubelet
```
This section describes how to enable the vSphere cloud provider. You will need to use the `cloud_provider` directive in the cluster YAML file.
### Related Links
- [vSphere Storage for Kubernetes](https://vmware.github.io/vsphere-storage-for-kubernetes/documentation/)
- [Kubernetes Persistent Volumes](https://kubernetes.io/docs/concepts/storage/persistent-volumes/)
- **Configuration:** For details on vSphere configuration in RKE, refer to the [configuration reference.]({{<baseurl>}}/rke/latest/en/config-options/cloud-providers/vsphere/config-reference)
- **Troubleshooting:** For guidance on troubleshooting a cluster with the vSphere cloud provider enabled, refer to the [troubleshooting section.]({{<baseurl>}}/rke/latest/en/config-options/cloud-providers/vsphere/troubleshooting)
- **Storage:** If you are setting up storage, see the [official vSphere documentation on storage for Kubernetes,](https://vmware.github.io/vsphere-storage-for-kubernetes/documentation/) or the [official Kubernetes documentation on persistent volumes.](https://kubernetes.io/docs/concepts/storage/persistent-volumes/) If you are using Rancher, refer to the [Rancher documentation on provisioning storage in vSphere.]({{<baseurl>}}/rancher/v2.x/en/cluster-admin/volumes-and-storage/examples/vsphere)
- **For Rancher users:** Refer to the Rancher documentation on [creating vSphere Kubernetes clusters]({{<baseurl>}}/rancher/v2.x/en/cluster-provisioning/rke-clusters/node-pools/vsphere) and [provisioning storage.]({{<baseurl>}}/rancher/v2.x/en/cluster-admin/volumes-and-storage/examples/vsphere)
# Prerequisites
- **Credentials:** You'll need to have credentials of a vCenter/ESXi user account with privileges allowing the cloud provider to interact with the vSphere infrastructure to provision storage. Refer to [this document](https://vmware.github.io/vsphere-storage-for-kubernetes/documentation/vcp-roles.html) to create and assign a role with the required permissions in vCenter.
- **VMware Tools** must be running in the Guest OS for all nodes in the cluster.
- **Disk UUIDs:** All nodes must be configured with disk UUIDs. This is required so that attached VMDKs present a consistent UUID to the VM, allowing the disk to be mounted properly. See the section on [enabling disk UUIDs.]({{<baseurl>}}/rke/latest/en/config-options/cloud-providers/vsphere/enabling-uuid)
# Enabling the vSphere Provider with the RKE CLI
To enable the vSphere Cloud Provider in the cluster, you must add the top-level `cloud_provider` directive to the cluster configuration file, set the `name` property to `vsphere` and add the `vsphereCloudProvider` directive containing the configuration matching your infrastructure. See the [configuration reference]({{<baseurl>}}/rke/latest/en/config-options/cloud-providers/vsphere/config-reference) for the gory details.
@@ -0,0 +1,144 @@
---
title: vSphere Configuration Reference
weight: 3
---
This section shows an example of how to configure the vSphere cloud provider.
The vSphere cloud provider must be enabled to allow dynamic provisioning of volumes.
For more details on deploying a Kubernetes cluster on vSphere, refer to the [official cloud provider documentation.](https://cloud-provider-vsphere.sigs.k8s.io/tutorials/kubernetes-on-vsphere-with-kubeadm.html)
> **Note:** This documentation reflects the new vSphere Cloud Provider configuration schema introduced in Kubernetes v1.9 which differs from previous versions.
# vSphere Configuration Example
Given the following:
- VMs in the cluster are running in the same datacenter `eu-west-1` managed by the vCenter `vc.example.com`.
- The vCenter has a user `provisioner` with password `secret` with the required roles assigned, see [Prerequisites](#prerequisites).
- The vCenter has a datastore named `ds-1` which should be used to store the VMDKs for volumes.
- A `vm/kubernetes` folder exists in vCenter.
The corresponding configuration for the provider would then be as follows:
```yaml
(...)
cloud_provider:
name: vsphere
vsphereCloudProvider:
virtual_center:
vc.example.com:
user: provisioner
password: secret
port: 443
datacenters: /us-west-1
workspace:
server: vc.example.com
folder: /us-west-1/folder/myvmfolder
default-datastore: /us-west-1/datastore/ds-1
datacenter: /us-west-1
resourcepool-path: /us-west-1/host/hn1/resources/myresourcepool
```
# Configuration Options
The vSphere configuration options are divided into 5 groups:
* [global](#global)
* [virtual_center](#virtual_center)
* [workspace](#workspace)
* [disk](#disk)
* [network](#network)
### global
The main purpose of global options is to be able to define a common set of configuration parameters that will be inherited by all vCenters defined under the `virtual_center` directive unless explicitly defined there.
Accordingly, the `global` directive accepts the same configuration options that are available under the `virtual_center` directive. Additionally it accepts a single parameter that can only be specified here:
| global Options | Type | Required | Description |
|:---------------:|:-------:|:---------:|:---------|
| insecure-flag | boolean | | Set to **true** if the vCenter/ESXi uses a self-signed certificate. |
**Example:**
```yaml
(...)
global:
insecure-flag: true
```
### virtual_center
This configuration directive specifies the vCenters that are managing the nodes in the cluster. You must define at least one vCenter/ESXi server. If the nodes span multiple vCenters then all must be defined.
Each vCenter is defined by adding a new entry under the `virtual_center` directive with the vCenter IP or FQDN as the name. All required parameters must be provided for each vCenter unless they are already defined under the `global` directive.
| virtual_center Options | Type | Required | Description |
|:----------------------:|:--------:|:---------:|:-----------|
| user | string | * | vCenter/ESXi user used to authenticate with this server. |
| password | string | * | User's password. |
| port | string | | Port to use to connect to this server. Defaults to 443. |
| datacenters | string | * | Comma-separated list of all datacenters in which cluster nodes are running in. |
| soap-roundtrip-count | uint | | Round tripper count for API requests to the vCenter (num retries = value - 1). |
> The following additional options (introduced in Kubernetes v1.11) are not yet supported in RKE.
| virtual_center Options | Type | Required | Description |
|:----------------------:|:--------:|:---------:|:-------|
| secret-name | string | | Name of secret resource containing credential key/value pairs. Can be specified in lieu of user/password parameters.|
| secret-namespace | string | | Namespace in which the secret resource was created in. |
| ca-file | string | | Path to CA cert file used to verify the vCenter certificate. |
**Example:**
```yaml
(...)
virtual_center:
172.158.111.1: {} # This vCenter inherits all it's properties from global options
172.158.110.2: # All required options are set explicitly
user: vc-user
password: othersecret
datacenters: eu-west-2
```
### workspace
This configuration group specifies how storage for volumes is created in vSphere.
The following configuration options are available:
| workspace Options | Type | Required | Description |
|:----------------------:|:--------:|:---------:|:---------|
| server | string | * | IP or FQDN of the vCenter/ESXi that should be used for creating the volumes. Must match one of the vCenters defined under the `virtual_center` directive.|
| datacenter | string | * | Name of the datacenter that should be used for creating volumes. For ESXi enter *ha-datacenter*.|
| folder | string | * | Path of folder in which to create dummy VMs used for volume provisioning (relative from the root folder in vCenter), e.g. "vm/kubernetes".|
| default-datastore | string | | Name of default datastore to place VMDKs if neither datastore or storage policy are specified in the volume options of a PVC. If datastore is located in a storage folder or is a member of a datastore cluster, specify the full path. |
| resourcepool-path | string | | Absolute or relative path to the resource pool where the dummy VMs for [Storage policy based provisioning](https://vmware.github.io/vsphere-storage-for-kubernetes/documentation/policy-based-mgmt.html) should be created. If a relative path is specified, it is resolved with respect to the datacenter's *host* folder. Examples: `/<dataCenter>/host/<hostOrClusterName>/Resources/<poolName>`, `Resources/<poolName>`. For standalone ESXi specify `Resources`. |
**Example:**
```yaml
(...)
workspace:
server: 172.158.111.1 # matches IP of vCenter defined in the virtual_center block
datacenter: eu-west-1
folder: vm/kubernetes
default-datastore: ds-1
```
### disk
The following configuration options are available under the disk directive:
| disk Options | Type | Required | Description |
|:--------------------:|:--------:|:---------:|:----------------|
| scsicontrollertype | string | | SCSI controller type to use when attaching block storage to VMs. Must be one of: *lsilogic-sas* or *pvscsi*. Default: *pvscsi*. |
### network
The following configuration options are available under the network directive:
| network Options | Type | Required | Description |
|:-------------------:|:--------:|:---------:|:-----------------------------------------------------------------------------|
| public-network | string | | Name of public **VM Network** to which the VMs in the cluster are connected. Used to determine public IP addresses of VMs.|
@@ -0,0 +1,35 @@
---
title: Enabling Disk UUIDs for vSphere VMs
weight: 2
---
In order to provision nodes with RKE, all nodes must be configured with disk UUIDs. This is required so that attached VMDKs present a consistent UUID to the VM, allowing the disk to be mounted properly.
Depending on whether you are provisioning the VMs using the [vSphere node driver]({{<baseurl>}}/rancher/v2.x/en/cluster-provisioning/rke-clusters/node-pools/vsphere) in Rancher or using your own scripts or third-party tools, there are different methods available to enable disk UUIDs for VMs:
- [Using the vSphere console](#using-the-vsphere-console)
- [Using the GOVC CLI tool](#using-the-govc-cli-tool)
- [Using a Rancher node template](#using-a-rancher-node-template)
### Using the vSphere Console
The required property can be set while creating or modifying VMs in the vSphere Console:
1. For each VM navigate to the tab **VM Options** and click on **Edit Configuration**.
2. Add the parameter `disk.EnableUUID` with a value of **TRUE**.
![vsphere-advanced-parameters]({{< baseurl >}}/img/rke/vsphere-advanced-parameters.png)
### Using the GOVC CLI tool
You can also modify properties of VMs with the [govc](https://github.com/vmware/govmomi/tree/master/govc) command-line tool to enable disk UUIDs:
```sh
$ govc vm.change -vm <vm-path> -e disk.enableUUID=TRUE
```
### Using a Rancher Node Template
In Rancher v2.0.4+, disk UUIDs are enabled in vSphere node templates by default.
If you are using Rancher prior to v2.0.4, refer to the [Rancher documentation.]({{<baseurl>}}/rancher/v2.x/en/cluster-provisioning/rke-clusters/node-pools/vsphere/#enabling-disk-uuids-with-a-node-template) for details on how to enable a UUID with a Rancher node template.
@@ -0,0 +1,27 @@
---
title: Troubleshooting vSphere Clusters
weight: 4
---
If you are experiencing issues while provisioning a cluster with enabled vSphere Cloud Provider or while creating vSphere volumes for your workloads, you should inspect the logs of the following K8s services:
- controller-manager (Manages volumes in vCenter)
- kubelet: (Mounts vSphere volumes to pods)
If your cluster is not configured with external [Cluster Logging]({{< baseurl >}}/rancher/v2.x/en/tools/logging/), you will need to SSH into nodes to get the logs of the `kube-controller-manager` (running on one of the control plane nodes) and the `kubelet` (pertaining to the node where the stateful pod has been scheduled).
The easiest way to create a SSH session with a node is the Rancher CLI tool.
1. [Configure the Rancher CLI]({{< baseurl >}}/rancher/v2.x/en/cli/) for your cluster.
2. Run the following command to get a shell to the corresponding nodes:
```sh
$ rancher ssh <nodeName>
```
3. Inspect the logs of the controller-manager and kubelet containers looking for errors related to the vSphere cloud provider:
```sh
$ docker logs --since 15m kube-controller-manager
$ docker logs --since 15m kubelet
```