mirror of
https://github.com/rancher/rancher-docs.git
synced 2026-09-25 20:48:11 +00:00
Merge pull request #547 from janeczku/vsphere-cloud-provider
vSphere cloud provider and storage docs
This commit is contained in:
Denise
@@ -9,4 +9,5 @@ Rancher supports persistent storage with a variety of volume plugins. However, b
|
||||
|
||||
For your convenience, Rancher offers documentation on how to configure some of the popular storage methods:
|
||||
|
||||
- [NFS Share](nfs)
|
||||
- [NFS]({{< baseurl >}}/rancher/v2.x/en/k8s-in-rancher/volumes-and-storage/examples/nfs/)
|
||||
- [vSphere]({{< baseurl >}}/rancher/v2.x/en/k8s-in-rancher/volumes-and-storage/examples/vsphere/)
|
||||
|
||||
@@ -0,0 +1,75 @@
|
||||
---
|
||||
title: vSphere Storage
|
||||
weight: 3055
|
||||
aliases:
|
||||
- /rancher/v2.x/en/tasks/clusters/adding-storage/provisioning-storage/vsphere/
|
||||
---
|
||||
|
||||
>**Prerequisites:**
|
||||
>
|
||||
>- Provisioning vSphere volumes requires the corresponding cloud provider to be enabled in the cluster, see [vSphere Cloud Provider]({{< baseurl >}}/rke/v0.1.x/en/config-options/cloud-providers/vsphere).
|
||||
|
||||
The recommended approach for providing vSphere storage to stateful workloads is creating a vSphereVolume [storage class]({{< baseurl >}}/rancher/v2.x/en/k8s-in-rancher/volumes-and-storage/#storage-classes). This allows for dynamic provisioning of vSphere storage when workloads request volumes trough a [persistent volume claim]({{< baseurl >}}/rancher/v2.x/en/k8s-in-rancher/volumes-and-storage/persistent-volume-claims/).
|
||||
|
||||
## Steps
|
||||
|
||||
Let's walk through the steps of creating the storage class and then deploy a stateful workload in the Rancher UI.
|
||||
|
||||
> **Note:**
|
||||
>
|
||||
> These steps can also be performed using the `kubectl` command line tool. See [Kubernetes documentation on persistent volumes](https://kubernetes.io/docs/concepts/storage/persistent-volumes/) for details.
|
||||
|
||||
### 1. Create Storage Class
|
||||
|
||||
Storage classes for vSphereVolumes may be created with a number of different properties. Refer to the [available options](https://vmware.github.io/vsphere-storage-for-kubernetes/documentation/storageclass.htm).
|
||||
|
||||
1. Navigate to **Storage** > **Storage Classes** while on the cluster level in the Rancher UI.
|
||||
2. Click on **Add Class**.
|
||||
3. Assign a **Name** and select **VMWare vSphere Volume** under **Provisioner**.
|
||||
|
||||
|
||||
|
||||
4. Click **Save**.
|
||||
|
||||
### 2. Create a workload with a vSphere volume
|
||||
|
||||
1. Navigate to a project in your cluster and under **Workloads** click on **Deploy**.
|
||||
2. Under **Workload Type** click **More Options** and select **Stateful set of 1 pod**.
|
||||
3. Assing a **Name** and Docker image.
|
||||
4. Expand the **Volumes** section and click **Add Volume**.
|
||||
5. Choose **Add a new persistent volume (claim)**. This will implicitly create the claim once you deploy the workload.
|
||||
6. Assign a **Name** for the claim, ie. *test-volume* and select the vSphere storage class created in the previous step.
|
||||
7. Enter the required **Capacity** for the volume. Then click **Define**.
|
||||
|
||||
|
||||
|
||||
8. Assign a path in the **Mount Point** field. This is the full path where the volume will be mounted in the container file system, e.g. */persistent*.
|
||||
9. Click **Launch** to create the workload.
|
||||
|
||||
### 3. Verify persistence of the volume
|
||||
|
||||
1. In the context menu for the created workload, click on **Execute Shell**.
|
||||
2. Note the directory in the root (in this case */persistent*) where the volume has been mounted to.
|
||||
3. Create a file in the volume by executing the command `touch /<volumeMountPoint>/data.txt`.
|
||||
4. **Close** the shell window.
|
||||
5. Click on the name of the workload to reveal detail information.
|
||||
6. Open the context menu next to the Pod in the *Running* state.
|
||||
7. Delete the Pod by selecting **Delete** and confirming the prompt.
|
||||
8. Observe how Kubernetes takes care of deleting it, then schedules a replacement pod in order to satsify the desired scale of the workload (= 1 replica).
|
||||
10. Once the replacement pod is running, open a shell to it.
|
||||
11. Inspect the content of the directory where the volume is mounted, ie. `ls -l /<volumeMountPoint>`. Note how the `file that was created in the now deleted pod is still present.
|
||||
|
||||
|
||||
|
||||
## Why you should use StatefulSets and not Deployments
|
||||
|
||||
Since vSphere volumes are backed by VMDK block storage they only support an [access mode](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#persistentvolumeclaims) of `ReadWriteOnce`, meaning that the volume can only be mounted to a single pod at a time, unless all pods consuming that volume are colocated on the same node. This makes a deployment resource unusable for scaling beyond a single replica if it consumes vSphere volumes.
|
||||
|
||||
Even using a deployment resource with just a single replica may result in a deadlock situation while updating the deployment: If the updated pod is scheduled to a node different from where the existing pod lives, it will fail to start because the VMDK is still attached to the other node.
|
||||
|
||||
That being said: You should always use [StatefulSets](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/) for workloads consuming vSphere storage as this resource type has been designed to handle the caveats of `ReadWriteOnce` volumes.
|
||||
|
||||
## Related Links
|
||||
|
||||
- [vSphere Storage for Kubernetes](https://vmware.github.io/vsphere-storage-for-kubernetes/documentation/)
|
||||
- [Kubernetes Persistent Volumes](https://kubernetes.io/docs/concepts/storage/persistent-volumes/)
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
title: Add-Ons
|
||||
weight: 250
|
||||
weight: 260
|
||||
---
|
||||
|
||||
RKE supports pluggable add-ons. Add-ons are used to deploy several cluster components including:
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
title: Ingress Controllers
|
||||
weight: 252
|
||||
weight: 262
|
||||
---
|
||||
|
||||
By default, RKE deploys the nginx ingress controller on all schedulable nodes.
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
title: Network Plug-ins
|
||||
weight: 251
|
||||
weight: 261
|
||||
---
|
||||
|
||||
RKE supports the following network plug-ins that are deployed as add-ons:
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
title: User-Defined Add-Ons
|
||||
weight: 253
|
||||
weight: 263
|
||||
---
|
||||
|
||||
Besides the [network plug-in]({{< baseurl >}}/rke/v0.1.x/en/config-options/network-plugins/) and [ingress controllers]({{< baseurl >}}/rke/v0.1.x/en/config-options/ingress-controllers/), you can define any add-on that you want deployed after the Kubernetes cluster is deployed.
|
||||
|
||||
@@ -1,385 +1,14 @@
|
||||
---
|
||||
title: Cloud Providers
|
||||
weight: 245
|
||||
weight: 250
|
||||
---
|
||||
|
||||
RKE supports the ability to set your specific [cloud provider](https://kubernetes.io/docs/concepts/cluster-administration/cloud-providers/) for your Kubernetes cluster. There are specific cloud configurations for these cloud providers:
|
||||
RKE supports the ability to set your specific [cloud provider](https://kubernetes.io/docs/concepts/cluster-administration/cloud-providers/) for your Kubernetes cluster. There are specific cloud configurations for these cloud providers.
|
||||
To enable a cloud provider its name as well as any required configuration options must be provided under the `cloud_provider` directive in the cluster YML.
|
||||
|
||||
* [AWS](#aws)
|
||||
* [Azure](#azure)
|
||||
* [OpenStack](#openstack)
|
||||
* [vSphere](#vsphere)
|
||||
* [AWS]({{< baseurl >}}/rke/v0.1.x/en/config-options/cloud-providers/aws)
|
||||
* [Azure]({{< baseurl >}}/rke/v0.1.x/en/config-options/cloud-providers/azure)
|
||||
* [OpenStack]({{< baseurl >}}/rke/v0.1.x/en/config-options/cloud-providers/openstack)
|
||||
* [vSphere]({{< baseurl >}}/rke/v0.1.x/en/config-options/cloud-providers/vsphere)
|
||||
|
||||
Outside of this list, RKE also supports the ability to handle any [custom cloud provider](#custom-cloud-provider).
|
||||
|
||||
The `cloud_provider` directive must be filled out with minimally a name. For each cloud provider, there are different configuration options to enable the cloud provider in Kubernetes.
|
||||
|
||||
### AWS
|
||||
|
||||
To enable the AWS cloud provider, there are no configuration options. You only need to set the name as `aws`. In order to use the AWS cloud provider, all cluster nodes must have already been configured with an appropriate IAM role.
|
||||
|
||||
```yaml
|
||||
cloud_provider:
|
||||
name: aws
|
||||
```
|
||||
|
||||
#### IAM Requirements
|
||||
|
||||
The nodes used in RKE that will be running the AWS cloud provider must have at least the following IAM policy.
|
||||
|
||||
```json
|
||||
{
|
||||
"Effect": "Allow",
|
||||
"Action": "ec2:Describe*",
|
||||
"Resource": "*"
|
||||
}
|
||||
```
|
||||
|
||||
In order to use Elastic Load Balancers (ELBs) and EBS with Kubernetes, the node(s) will need to have the an IAM role with appropriate access.
|
||||
|
||||
##### Example Policy for IAM Role:
|
||||
|
||||
```json
|
||||
{
|
||||
"Version": "2012-10-17",
|
||||
"Statement": [
|
||||
{
|
||||
"Effect": "Allow",
|
||||
"Action": "ec2:Describe*",
|
||||
"Resource": "*"
|
||||
},
|
||||
{
|
||||
"Effect": "Allow",
|
||||
"Action": "ec2:AttachVolume",
|
||||
"Resource": "*"
|
||||
},
|
||||
{
|
||||
"Effect": "Allow",
|
||||
"Action": "ec2:DetachVolume",
|
||||
"Resource": "*"
|
||||
},
|
||||
{
|
||||
"Effect": "Allow",
|
||||
"Action": ["elasticloadbalancing:*"],
|
||||
"Resource": ["*"]
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### Azure
|
||||
|
||||
To enable the Azure cloud provider, besides setting the name as `azure`, there are specific configuration options that must be set. Additionally, the Azure node name must also match the Kubernetes node name.
|
||||
|
||||
```yaml
|
||||
cloud_provider:
|
||||
name: azure
|
||||
azureCloudProvider:
|
||||
aadClientId: xxxxxxxxx
|
||||
aadClientSecret: xxxxxxxxx
|
||||
location: xxxxxxxxx
|
||||
resourceGroup: xxxxxxxxx
|
||||
subnetName: xxxxxxxxx
|
||||
subscriptionId: xxxxxxxxx
|
||||
vnetName: xxxxxxxxx
|
||||
tenantId: xxxxxxxxx
|
||||
securityGroupName: xxxxxxxxx
|
||||
```
|
||||
|
||||
#### Overriding the hostname
|
||||
|
||||
Since the Azure node name must match the Kubernetes node name, you override the Kubernetes name on the node by setting the `hostname_override` for each node. If you do not set the `hostname_override`, the Kubernetes node name will be set as the `address`, which will cause the Azure cloud provider to fail.
|
||||
|
||||
```yaml
|
||||
nodes:
|
||||
- address: x.x.x.x
|
||||
hostname_override: azure-rke1
|
||||
user: ubuntu
|
||||
role:
|
||||
- controlplane
|
||||
- etcd
|
||||
- worker
|
||||
```
|
||||
|
||||
#### Azure Configuration Options
|
||||
|
||||
Besides the minimum set of options, there are many other options that are supported in RKE:
|
||||
|
||||
| Azure Configuration Options | Type | Required |
|
||||
|:----------------------------: |:------: |:---------:|
|
||||
| tenantId | string | * |
|
||||
| subscriptionId | string | * |
|
||||
| aadClientId | string | * |
|
||||
| aadClientSecret | string | * |
|
||||
| cloud | string | |
|
||||
| resourceGroup | string | |
|
||||
| location | string | |
|
||||
| vnetName | string | |
|
||||
| vnetResourceGroup | string | |
|
||||
| subnetName | string | |
|
||||
| securityGroupName | string | |
|
||||
| routeTableName | string | |
|
||||
| primaryAvailabilitySetName | string | |
|
||||
| vmType | string | |
|
||||
| primaryScaleSetName | string | |
|
||||
| aadClientCertPath | string | |
|
||||
| aadClientCertPassword | string | |
|
||||
| cloudProviderBackoff | bool | |
|
||||
| cloudProviderBackoffRetries | int | |
|
||||
| cloudProviderBackoffExponent | int | |
|
||||
| cloudProviderBackoffDuration | int | |
|
||||
| cloudProviderBackoffJitter | int | |
|
||||
| cloudProviderRateLimit | bool | |
|
||||
| cloudProviderRateLimitQPS | int | |
|
||||
| cloudProviderRateLimitBucket | int | |
|
||||
| useInstanceMetadata | bool | |
|
||||
| useManagedIdentityExtension | bool | |
|
||||
| maximumLoadBalancerRuleCount | int | |
|
||||
|
||||
### Openstack
|
||||
|
||||
To enable the Openstack cloud provider, besides setting the name as `openstack`, there are specific configuration options that must be set. The Openstack configuration options are grouped into different sections.
|
||||
|
||||
```yaml
|
||||
cloud_provider:
|
||||
name: openstack
|
||||
openstackCloudProvider:
|
||||
global:
|
||||
username: xxxxxxxxxxxxxx
|
||||
password: xxxxxxxxxxxxxx
|
||||
auth-url: https://1.2.3.4/identity/v3
|
||||
tenant-id: xxxxxxxxxxxxxx
|
||||
domain-id: xxxxxxxxxxxxxx
|
||||
load_balancer:
|
||||
subnet-id: xxxxxxxxxxxxxx
|
||||
block_storage:
|
||||
ignore-volume-az: true
|
||||
route:
|
||||
router-id: xxxxxxxxxxxxxx
|
||||
metadata:
|
||||
search-order: xxxxxxxxxxxxxx
|
||||
```
|
||||
|
||||
#### Overriding the hostname
|
||||
|
||||
The OpenStack cloud provider uses the instance name (as determined from OpenStack metadata) as the name of the Kubernetes Node object, you must override the Kubernetes name on the node by setting the `hostname_override` for each node. If you do not set the `hostname_override`, the Kubernetes node name will be set as the `address`, which will cause the Openstack cloud provider to fail.
|
||||
|
||||
#### Openstack Configuration Options
|
||||
|
||||
The Openstack configuration options are divided into 5 groups.
|
||||
|
||||
* Global
|
||||
* Load Balancer
|
||||
* Block Storage
|
||||
* Route
|
||||
* Metadata
|
||||
|
||||
##### Global
|
||||
|
||||
These are the options that are available under the `global` directive.
|
||||
|
||||
| OpenStack's Global Configuration Options | Type | Required |
|
||||
|:--------------------: |:------: |:---------:|
|
||||
| auth_url | string | * |
|
||||
| username | string | * |
|
||||
| user-id | string | * |
|
||||
| password | string | * |
|
||||
| tenant-id | string | * |
|
||||
| tenant-name | string | |
|
||||
| trust-id | string | |
|
||||
| domain-id | string | |
|
||||
| domain-name | string | |
|
||||
| region | string | |
|
||||
| ca-file | string | |
|
||||
|
||||
##### Load Balancer
|
||||
|
||||
These are the options that are available under the `load_balancer` directive.
|
||||
|
||||
| OpenStack's Load Balancer Configuration Options | Type | Required |
|
||||
|:----------------------: |:------: |:---------:|
|
||||
| lb-version | string | |
|
||||
| use-octavia | bool | |
|
||||
| subnet-id | string | |
|
||||
| floating-network-id | string | |
|
||||
| lb-method | string | |
|
||||
| lb-provider | string | |
|
||||
| manage-security-groups | bool | |
|
||||
| create-monitor | bool | |
|
||||
| monitor-delay | int | * if `create-monitor` is true |
|
||||
| monitor-timeout | int | * if `create-monitor` is true |
|
||||
| monitor-max-retries | int | * if `create-monitor` is true |
|
||||
|
||||
|
||||
##### Block Storage
|
||||
|
||||
These are the options that are available under the `block_storage` directive.
|
||||
|
||||
| OpenStack's Block Storage Configuration Options | Type | Required |
|
||||
|:--------------------: |:------: |:---------:|
|
||||
| bs-version | string | |
|
||||
| trust-device-path | bool | |
|
||||
| ignore-volume-az | bool | |
|
||||
|
||||
##### Route
|
||||
|
||||
This is the option that is available under the `route` directive.
|
||||
|
||||
| OpenStack's Route Configuration Option | Type | Required |
|
||||
|:--------------------: |:------: |:---------:|
|
||||
| router-id | string | |
|
||||
|
||||
##### Metadata
|
||||
|
||||
These are the options that are available under the `metadata` directive.
|
||||
|
||||
| OpenStack's Metadata Configuration Options | Type | Required |
|
||||
|:--------------------: |:------: |:---------:|
|
||||
| search-order | string | |
|
||||
| request-timeout | int | |
|
||||
|
||||
For more information of Openstack configurations options please refer to the official Kubernetes [documentation](https://kubernetes.io/docs/concepts/cluster-administration/cloud-providers/#openstack).
|
||||
|
||||
### vSphere
|
||||
|
||||
To enable the vSphere cloud provider, besides setting the name as `vsphere`, there are specific configuration options that must be set. The vSphere configuration options are grouped into different sections.
|
||||
|
||||
```yaml
|
||||
cloud_provider:
|
||||
name: vsphere
|
||||
vsphereCloudProvider:
|
||||
global:
|
||||
user: user
|
||||
password: pass
|
||||
server: 1.2.3.4
|
||||
port: 22
|
||||
virtual_center:
|
||||
1.2.3.4:
|
||||
user: test
|
||||
password: test
|
||||
port: test
|
||||
5.6.7.8:
|
||||
user: test
|
||||
password: test
|
||||
port: test
|
||||
workspace:
|
||||
server: test.test.com
|
||||
datacenter: test
|
||||
folder: test
|
||||
default-datastore: test
|
||||
resourcepool-path: test
|
||||
network:
|
||||
public-network: xxxxxxxxxxxxxx
|
||||
disk:
|
||||
scsicontrollertype: xxxxxxxxxxxxxx
|
||||
```
|
||||
|
||||
#### vSphere Configuration Options
|
||||
|
||||
The vSphere configuration options are divided into 5 groups.
|
||||
|
||||
* Global
|
||||
* Virtual Center
|
||||
* Workspace
|
||||
* Network
|
||||
* Disk
|
||||
|
||||
##### Global
|
||||
|
||||
These are the options that are available under the `global` directive.
|
||||
|
||||
| vSphere's Global Configuration Options | Type | Required |
|
||||
|:--------------------: |:------: |:---------:|
|
||||
| user | string | |
|
||||
| password | string | |
|
||||
| server | string | |
|
||||
| port | string | |
|
||||
| insecure-flag | bool | |
|
||||
| datacenter | string | |
|
||||
| datacenters | string | |
|
||||
| datastore | string | |
|
||||
| working-dir | string | |
|
||||
| soap-roundtrip-count | int | |
|
||||
| vm-uuid | string | |
|
||||
|vm-name |string | |
|
||||
|
||||
|
||||
##### Virtual Center
|
||||
|
||||
These are the options that are available under `virtual_center`, which is a dictionary of vCenters. Each vCenter is defined individually.
|
||||
|
||||
```yaml
|
||||
virtual_center:
|
||||
<vcenter1-ip>:
|
||||
user: test
|
||||
password: test
|
||||
port: test
|
||||
<vcenter2-ip>:
|
||||
user: test
|
||||
password: test
|
||||
port: test
|
||||
```
|
||||
|
||||
For each `virtual_center`, these are the available configuration options to be used under the each virtual center. The virtual center's are separated from each other based on their IP.
|
||||
|
||||
| vSphere's Virtual Center Configuration Options | Type | Required |
|
||||
|:--------------------: |:------: |:---------:|
|
||||
| user | string | * if `global` `user` is not set |
|
||||
| password | string | * if `global` `password` is not set |
|
||||
| port | string | |
|
||||
| datacenters | string | |
|
||||
| soap-roundtrip-count | int | |
|
||||
|
||||
##### Workspace
|
||||
|
||||
These are the options that are available under the `workspace` directive.
|
||||
|
||||
| vSphere's Workspace Configuration Options | Type | Required |
|
||||
|:--------------------: |:------: |:---------:|
|
||||
| server | string | * |
|
||||
| datacenter | string | * |
|
||||
| folder | string | * |
|
||||
| default-datastore | string | * |
|
||||
| resourcepool-path | string | * |
|
||||
|
||||
##### Network
|
||||
|
||||
This is the option that is available under the `network` directive.
|
||||
|
||||
| vSphere's Network Configuration Option | Type | Required |
|
||||
|:--------------------: |:------: |:---------:|
|
||||
| public-network | string | |
|
||||
|
||||
##### Disk
|
||||
|
||||
This is the option that is available under the `disk` directive.
|
||||
|
||||
| vSphere's Disk Configuration Option | Type | Required |
|
||||
|:--------------------: |:------: |:---------:|
|
||||
| scsicontrollertype | string | |
|
||||
|
||||
### Custom Cloud Provider
|
||||
|
||||
If you want to enable a different cloud provider, RKE allows for custom cloud provider options. A name must be provided and the custom Cloud Provider options can be passed in as a multiline string in `customCloudProvider`.
|
||||
|
||||
For example, in order to use the oVirt cloud provider with Kubernetes, here's the following cloud provider information:
|
||||
|
||||
```
|
||||
[connection]
|
||||
uri = https://localhost:8443/ovirt-engine/api
|
||||
username = admin@internal
|
||||
password = admin
|
||||
```
|
||||
|
||||
To add this cloud config file to RKE, the `cloud_provider` would be need to be set.
|
||||
|
||||
```yaml
|
||||
cloud_provider:
|
||||
name: ovirt
|
||||
# Note the pipe as this is what indicates a multiline string
|
||||
customCloudProvider: |-
|
||||
[connection]
|
||||
uri = https://localhost:8443/ovirt-engine/api
|
||||
username = admin@internal
|
||||
password = admin
|
||||
```
|
||||
Outside of this list, RKE also supports the ability to handle any [custom cloud provider]({{< baseurl >}}/rke/v0.1.x/en/config-options/cloud-providers/custom).
|
||||
|
||||
@@ -0,0 +1,55 @@
|
||||
---
|
||||
title: AWS Cloud Provider
|
||||
weight: 251
|
||||
---
|
||||
|
||||
To enable the AWS cloud provider, there are no configuration options. You only need to set the name as `aws`. In order to use the AWS cloud provider, all cluster nodes must have already been configured with an appropriate IAM role.
|
||||
|
||||
```yaml
|
||||
cloud_provider:
|
||||
name: aws
|
||||
```
|
||||
|
||||
## IAM Requirements
|
||||
|
||||
The nodes used in RKE that will be running the AWS cloud provider must have at least the following IAM policy.
|
||||
|
||||
```json
|
||||
{
|
||||
"Effect": "Allow",
|
||||
"Action": "ec2:Describe*",
|
||||
"Resource": "*"
|
||||
}
|
||||
```
|
||||
|
||||
In order to use Elastic Load Balancers (ELBs) and EBS with Kubernetes, the node(s) will need to have the an IAM role with appropriate access.
|
||||
|
||||
## Example Policy for IAM Role:
|
||||
|
||||
```json
|
||||
{
|
||||
"Version": "2012-10-17",
|
||||
"Statement": [
|
||||
{
|
||||
"Effect": "Allow",
|
||||
"Action": "ec2:Describe*",
|
||||
"Resource": "*"
|
||||
},
|
||||
{
|
||||
"Effect": "Allow",
|
||||
"Action": "ec2:AttachVolume",
|
||||
"Resource": "*"
|
||||
},
|
||||
{
|
||||
"Effect": "Allow",
|
||||
"Action": "ec2:DetachVolume",
|
||||
"Resource": "*"
|
||||
},
|
||||
{
|
||||
"Effect": "Allow",
|
||||
"Action": ["elasticloadbalancing:*"],
|
||||
"Resource": ["*"]
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
@@ -0,0 +1,71 @@
|
||||
---
|
||||
title: Azure Cloud Provider
|
||||
weight: 252
|
||||
---
|
||||
|
||||
To enable the Azure cloud provider, besides setting the name as `azure`, there are specific configuration options that must be set. Additionally, the Azure node name must also match the Kubernetes node name.
|
||||
|
||||
```yaml
|
||||
cloud_provider:
|
||||
name: azure
|
||||
azureCloudProvider:
|
||||
aadClientId: xxxxxxxxx
|
||||
aadClientSecret: xxxxxxxxx
|
||||
location: xxxxxxxxx
|
||||
resourceGroup: xxxxxxxxx
|
||||
subnetName: xxxxxxxxx
|
||||
subscriptionId: xxxxxxxxx
|
||||
vnetName: xxxxxxxxx
|
||||
tenantId: xxxxxxxxx
|
||||
securityGroupName: xxxxxxxxx
|
||||
```
|
||||
|
||||
## Overriding the hostname
|
||||
|
||||
Since the Azure node name must match the Kubernetes node name, you override the Kubernetes name on the node by setting the `hostname_override` for each node. If you do not set the `hostname_override`, the Kubernetes node name will be set as the `address`, which will cause the Azure cloud provider to fail.
|
||||
|
||||
```yaml
|
||||
nodes:
|
||||
- address: x.x.x.x
|
||||
hostname_override: azure-rke1
|
||||
user: ubuntu
|
||||
role:
|
||||
- controlplane
|
||||
- etcd
|
||||
- worker
|
||||
```
|
||||
|
||||
## Azure Configuration Options
|
||||
|
||||
Besides the minimum set of options, there are many other options that are supported in RKE:
|
||||
|
||||
| Azure Configuration Options | Type | Required |
|
||||
|:----------------------------: |:------: |:---------:|
|
||||
| tenantId | string | * |
|
||||
| subscriptionId | string | * |
|
||||
| aadClientId | string | * |
|
||||
| aadClientSecret | string | * |
|
||||
| cloud | string | |
|
||||
| resourceGroup | string | |
|
||||
| location | string | |
|
||||
| vnetName | string | |
|
||||
| vnetResourceGroup | string | |
|
||||
| subnetName | string | |
|
||||
| securityGroupName | string | |
|
||||
| routeTableName | string | |
|
||||
| primaryAvailabilitySetName | string | |
|
||||
| vmType | string | |
|
||||
| primaryScaleSetName | string | |
|
||||
| aadClientCertPath | string | |
|
||||
| aadClientCertPassword | string | |
|
||||
| cloudProviderBackoff | bool | |
|
||||
| cloudProviderBackoffRetries | int | |
|
||||
| cloudProviderBackoffExponent | int | |
|
||||
| cloudProviderBackoffDuration | int | |
|
||||
| cloudProviderBackoffJitter | int | |
|
||||
| cloudProviderRateLimit | bool | |
|
||||
| cloudProviderRateLimitQPS | int | |
|
||||
| cloudProviderRateLimitBucket | int | |
|
||||
| useInstanceMetadata | bool | |
|
||||
| useManagedIdentityExtension | bool | |
|
||||
| maximumLoadBalancerRuleCount | int | |
|
||||
@@ -0,0 +1,28 @@
|
||||
---
|
||||
title: Custom Cloud Provider
|
||||
weight: 255
|
||||
---
|
||||
|
||||
If you want to enable a different cloud provider, RKE allows for custom cloud provider options. A name must be provided and the custom Cloud Provider options can be passed in as a multiline string in `customCloudProvider`.
|
||||
|
||||
For example, in order to use the oVirt cloud provider with Kubernetes, here's the following cloud provider information:
|
||||
|
||||
```
|
||||
[connection]
|
||||
uri = https://localhost:8443/ovirt-engine/api
|
||||
username = admin@internal
|
||||
password = admin
|
||||
```
|
||||
|
||||
To add this cloud config file to RKE, the `cloud_provider` would be need to be set.
|
||||
|
||||
```yaml
|
||||
cloud_provider:
|
||||
name: ovirt
|
||||
# Note the pipe as this is what indicates a multiline string
|
||||
customCloudProvider: |-
|
||||
[connection]
|
||||
uri = https://localhost:8443/ovirt-engine/api
|
||||
username = admin@internal
|
||||
password = admin
|
||||
```
|
||||
@@ -0,0 +1,106 @@
|
||||
---
|
||||
title: Openstack Cloud Provider
|
||||
weight: 253
|
||||
---
|
||||
|
||||
To enable the Openstack cloud provider, besides setting the name as `openstack`, there are specific configuration options that must be set. The Openstack configuration options are grouped into different sections.
|
||||
|
||||
```yaml
|
||||
cloud_provider:
|
||||
name: openstack
|
||||
openstackCloudProvider:
|
||||
global:
|
||||
username: xxxxxxxxxxxxxx
|
||||
password: xxxxxxxxxxxxxx
|
||||
auth-url: https://1.2.3.4/identity/v3
|
||||
tenant-id: xxxxxxxxxxxxxx
|
||||
domain-id: xxxxxxxxxxxxxx
|
||||
load_balancer:
|
||||
subnet-id: xxxxxxxxxxxxxx
|
||||
block_storage:
|
||||
ignore-volume-az: true
|
||||
route:
|
||||
router-id: xxxxxxxxxxxxxx
|
||||
metadata:
|
||||
search-order: xxxxxxxxxxxxxx
|
||||
```
|
||||
|
||||
## Overriding the hostname
|
||||
|
||||
The OpenStack cloud provider uses the instance name (as determined from OpenStack metadata) as the name of the Kubernetes Node object, you must override the Kubernetes name on the node by setting the `hostname_override` for each node. If you do not set the `hostname_override`, the Kubernetes node name will be set as the `address`, which will cause the Openstack cloud provider to fail.
|
||||
|
||||
## Openstack Configuration Options
|
||||
|
||||
The Openstack configuration options are divided into 5 groups.
|
||||
|
||||
* Global
|
||||
* Load Balancer
|
||||
* Block Storage
|
||||
* Route
|
||||
* Metadata
|
||||
|
||||
### Global
|
||||
|
||||
These are the options that are available under the `global` directive.
|
||||
|
||||
| OpenStack's Global Configuration Options | Type | Required |
|
||||
|:--------------------: |:------: |:---------:|
|
||||
| auth_url | string | * |
|
||||
| username | string | * |
|
||||
| user-id | string | * |
|
||||
| password | string | * |
|
||||
| tenant-id | string | * |
|
||||
| tenant-name | string | |
|
||||
| trust-id | string | |
|
||||
| domain-id | string | |
|
||||
| domain-name | string | |
|
||||
| region | string | |
|
||||
| ca-file | string | |
|
||||
|
||||
### Load Balancer
|
||||
|
||||
These are the options that are available under the `load_balancer` directive.
|
||||
|
||||
| OpenStack's Load Balancer Configuration Options | Type | Required |
|
||||
|:----------------------: |:------: |:---------:|
|
||||
| lb-version | string | |
|
||||
| use-octavia | bool | |
|
||||
| subnet-id | string | |
|
||||
| floating-network-id | string | |
|
||||
| lb-method | string | |
|
||||
| lb-provider | string | |
|
||||
| manage-security-groups | bool | |
|
||||
| create-monitor | bool | |
|
||||
| monitor-delay | int | * if `create-monitor` is true |
|
||||
| monitor-timeout | int | * if `create-monitor` is true |
|
||||
| monitor-max-retries | int | * if `create-monitor` is true |
|
||||
|
||||
|
||||
### Block Storage
|
||||
|
||||
These are the options that are available under the `block_storage` directive.
|
||||
|
||||
| OpenStack's Block Storage Configuration Options | Type | Required |
|
||||
|:--------------------: |:------: |:---------:|
|
||||
| bs-version | string | |
|
||||
| trust-device-path | bool | |
|
||||
| ignore-volume-az | bool | |
|
||||
|
||||
### Route
|
||||
|
||||
This is the option that is available under the `route` directive.
|
||||
|
||||
| OpenStack's Route Configuration Option | Type | Required |
|
||||
|:--------------------: |:------: |:---------:|
|
||||
| router-id | string | |
|
||||
|
||||
### Metadata
|
||||
|
||||
These are the options that are available under the `metadata` directive.
|
||||
|
||||
| OpenStack's Metadata Configuration Options | Type | Required |
|
||||
|:--------------------: |:------: |:---------:|
|
||||
| search-order | string | |
|
||||
| request-timeout | int | |
|
||||
|
||||
For more information of Openstack configurations options please refer to the official Kubernetes [documentation](https://kubernetes.io/docs/concepts/cluster-administration/cloud-providers/#openstack).
|
||||
@@ -0,0 +1,258 @@
|
||||
---
|
||||
title: vSphere Cloud Provider
|
||||
weight: 254
|
||||
---
|
||||
|
||||
The [vSphere Cloud Provider](https://vmware.github.io/vsphere-storage-for-kubernetes/documentation/) interacts with VMware infrastructure (vCenter or standalone ESXi server) to provision and manage storage for persistent volumes in a Kubernetes cluster.
|
||||
|
||||
When provisioning Kubernetes using RKE CLI or using [RKE clusters]({{< baseurl >}}/rancher/v2.x/en/cluster-provisioning/rke-clusters/) in Rancher, the vSphere Cloud Provider can be enabled by configuring the `cloud_provider` directive in the cluster YAML file.
|
||||
|
||||
### Prerequisites
|
||||
|
||||
1. You'll need to have credentials of a vCenter/ESXi user account with privileges allowing the cloud provider to interact with the vSphere infrastructure to provision storage. Refer to [this document](https://vmware.github.io/vsphere-storage-for-kubernetes/documentation/vcp-roles.html) to create and assign a role with the required permissions in vCenter.
|
||||
2. VMware Tools must be running in the Guest OS for all nodes in the cluster.
|
||||
3. All nodes must be configured with disk UUIDs. This is required so that attached VMDKs present a consistent UUID to the VM, allowing the disk to be mounted properly. See [Enabling Disk UUIDs](#enabling-disk-uuids-for-vsphere-vms).
|
||||
|
||||
## Clusters provisioned with RKE CLI
|
||||
|
||||
To enable the vSphere Cloud Provider in the cluster, you must add the top-level `cloud_provider` directive to the cluster configuration file, set the `name` property to `vsphere` and add the `vsphereCloudProvider` directive containing the configuration matching your infrastructure. See the [configuration reference](#configuration-reference) for the gory details.
|
||||
|
||||
## Clusters provisioned with Rancher
|
||||
|
||||
When provisioning clusters in Rancher using the [vSphere node driver]({{< baseurl >}}/rancher/v2.x/en/cluster-provisioning/rke-clusters/node-pools/vsphere/) or on pre-created [custom nodes]({{< baseurl >}}/rancher/v2.x/en/cluster-provisioning/rke-clusters/custom-nodes/) the cluster YAML file must be modified in order to enable the cloud provider.
|
||||
|
||||
1. Log in to the Rancher UI as admin user.
|
||||
2. Navigate to **Clusters** in the **Global** view.
|
||||
3. Click **Add Cluster** and select the **vSphere** infrastructure provider.
|
||||
4. Assign a **Cluster Name**.
|
||||
5. Assign **Member Roles** as required.
|
||||
6. Expand **Cluster Options** and configure as required.
|
||||
7. Set **Cloud Provider** option to `Custom`.
|
||||
|
||||
|
||||
|
||||
8. Click on **Edit as YAML**
|
||||
9. Insert the following top-level structure to the pre-populated cluster YAML. Note that the `name` *must* be set to `vsphere`. Refer to the [configuration reference](#configuration-reference) to learn about the properties of the `vsphereCloudProvider` directive.
|
||||
|
||||
```yaml
|
||||
cloud_provider:
|
||||
name: vsphere
|
||||
vsphereCloudProvider:
|
||||
[Insert provider configuration]
|
||||
```
|
||||
|
||||
10. Configure the **Node Pools** per your requirements while ensuring to use a node template that enables disk UUIDs for the VMs (See [Annex - Enable disk UUIDs for vSphere VMs]).
|
||||
11. Click on **Create** to start provisioning the VMs and Kubernetes services.
|
||||
|
||||
## Configuration Reference
|
||||
|
||||
> **Note:** This documentation reflects the new vSphere Cloud Provider configuration schema introduced in Kubernetes v1.9 which differs from previous versions.
|
||||
|
||||
The vSphere configuration options are divided into 5 groups:
|
||||
|
||||
* global
|
||||
* virtual_center
|
||||
* workspace
|
||||
* disk
|
||||
* network
|
||||
|
||||
### global
|
||||
|
||||
The main purpose of global options is to be able to define a common set of configuration parameters that will be inherited by all vCenters defined under the `virtual_center` directive unless explicitely defined there.
|
||||
|
||||
Accordingly, the `global` directive accepts the same configuration options that are available under the `virtual_center` directive. Additionally it accepts a single parameter that can only be specified here:
|
||||
|
||||
| global Options | Type | Required | Description |
|
||||
|:---------------:|:-------:|:---------:|:-----------------------------------------------------------------------------:|
|
||||
| insecure-flag | boolean | | Set to **true** if the vCenter/ESXi uses a self-signed certificate. |
|
||||
|
||||
___
|
||||
|
||||
**Example:**
|
||||
|
||||
```yaml
|
||||
(...)
|
||||
global:
|
||||
insecure-flag: true
|
||||
```
|
||||
|
||||
### virtual_center
|
||||
|
||||
This configuration directive specifies the vCenters that are managing the nodes in the cluster. You must define at least one vCenter/ESXi server. If the nodes span multiple vCenters then all must be defined.
|
||||
|
||||
Each vCenter is defined by adding a new entry under the `virtual_center` directive with the vCenter IP or FQDN as the name. All required parameters must be provided for each vCenter unless they are already defined under the `global` directive.
|
||||
|
||||
| virtual_center Options | Type | Required | Description |
|
||||
|:----------------------:|:--------:|:---------:|:-----------------------------------------------------------------------------:|
|
||||
| user | string | * | vCenter/ESXi user used to authenticate with this server. |
|
||||
| password | string | * | User's password. |
|
||||
| port | string | | Port to use to connect to this server. Defaults to 443. |
|
||||
| datacenters | string | * | Comma-separated list of all datacenters in which cluster nodes are running in.|
|
||||
| soap-roundtrip-count | uint | | Round tripper count for API requests to the vCenter (num retries = value - 1).|
|
||||
|
||||
|
||||
> The following additional options (introduced in Kubernetes v1.11) are not yet supported in RKE.
|
||||
|
||||
| virtual_center Options | Type | Required | Description |
|
||||
|:----------------------:|:--------:|:---------:|:-----------------------------------------------------------------------------:|
|
||||
| secret-name | string | | Name of secret resource containing credential key/value pairs. Can be specified in lieu of user/password parameters.|
|
||||
| secret-namespace | string | | Namespace in which the secret resource was created in. |
|
||||
| ca-file | string | | Path to CA cert file used to verify the vCenter certificate. |
|
||||
|
||||
___
|
||||
|
||||
**Example:**
|
||||
|
||||
```yaml
|
||||
(...)
|
||||
virtual_center:
|
||||
172.158.111.1: {} # This vCenter inherits all it's properties from global options
|
||||
172.158.110.2: # All required options are set explicitly
|
||||
user: vc-user
|
||||
password: othersecret
|
||||
datacenters: eu-west-2
|
||||
```
|
||||
|
||||
### workspace
|
||||
|
||||
This configuration group specifies how storage for volumes is created in vSphere.
|
||||
The following configuration options are available:
|
||||
|
||||
| workspace Options | Type | Required | Description |
|
||||
|:----------------------:|:--------:|:---------:|:-----------------------------------------------------------------------------:|
|
||||
| server | string | * | IP or FQDN of the vCenter/ESXi that should be used for creating the volumes. Must match one of the vCenters defined under the `virtual_center` directive.|
|
||||
| datacenter | string | * | Name of the datacenter that should be used for creating volumes. For ESXi enter *ha-datacenter*.|
|
||||
| folder | string | * | Path of folder in which to create dummy VMs used for volume provisioning (relative from the root of the datastore).|
|
||||
| default-datastore | string | | Name of datastore used for volumes if neither datastore or storage policy are specified in the volume options of a PVC.|
|
||||
| resourcepool-path | string | | Path of the vSphere resource pool that should be used to create dummy VMs. Applies only to policy-based provisioning of vSphere volumes. See [Storage policy based provisioning](https://vmware.github.io/vsphere-storage-for-kubernetes/documentation/policy-based-mgmt.html).|
|
||||
|
||||
___
|
||||
|
||||
**Example:**
|
||||
|
||||
```yaml
|
||||
(...)
|
||||
workspace:
|
||||
server: 172.158.111.1 # matches IP of vCenter defined in the virtual_center block
|
||||
datacenter: eu-west-1
|
||||
folder: k8s-dummy
|
||||
default-datastore: ds-1
|
||||
```
|
||||
|
||||
### disk
|
||||
|
||||
The following configuration options are available under the disk directive:
|
||||
|
||||
| disk Options | Type | Required | Description |
|
||||
|:--------------------:|:--------:|:---------:|:-----------------------------------------------------------------------------:|
|
||||
| scsicontrollertype | string | | SCSI controller type to use when attaching block storage to VMs. Must be one of: *lsilogic-sas* or *pvscsi*. Default: *pvscsi*.|
|
||||
|
||||
___
|
||||
|
||||
### network
|
||||
|
||||
The following configuration options are available under the network directive:
|
||||
|
||||
| network Options | Type | Required | Description |
|
||||
|:-------------------:|:--------:|:---------:|:-----------------------------------------------------------------------------:|
|
||||
| public-network | string | | Name of public **VM Network** to which the VMs in the cluster are connected. Used to determine public IP addresses of VMs.|
|
||||
|
||||
|
||||
## Configuration Example
|
||||
|
||||
Given the following:
|
||||
|
||||
- VMs in the cluster are running in the same datacenter `eu-west-1` managed by the vCenter `vc.example.com`.
|
||||
- The vCenter has a user `provisioner` with password `secret` with the required roles assigned, see [Prerequisites](#prerequisites).
|
||||
- The vCenter has a datastore named `ds-1` which should be used to store the VMDKs for volumes.
|
||||
- A `k8s-dummy` folder exists in the root of the datastore.
|
||||
|
||||
The corresponding configuration for the provider would then be as follows:
|
||||
|
||||
```yaml
|
||||
(...)
|
||||
cloud_provider:
|
||||
name: vsphere
|
||||
vsphereCloudProvider:
|
||||
virtual_center:
|
||||
vc.example.com:
|
||||
user: provisioner
|
||||
password: secret
|
||||
datacenters: eu-west-1
|
||||
workspace:
|
||||
server: vc.example.com
|
||||
folder: k8s-dummy
|
||||
default-datastore: ds-1
|
||||
datacenter: eu-west-1
|
||||
|
||||
```
|
||||
|
||||
## Annex
|
||||
|
||||
### Enabling disk UUIDs for vSphere VMs
|
||||
|
||||
Depending on whether you are provisioning the VMs using the [vSphere node driver]({{< baseurl >}}/rancher/v2.x/en/cluster-provisioning/rke-clusters/node-pools/vsphere) in Rancher or using your own scripts or third-party tools, there are different methods available to enable disk UUIDs for VMs.
|
||||
|
||||
#### Using the Vsphere Console
|
||||
|
||||
The required property can be set while creating or modifying VMs in the vSphere Console:
|
||||
|
||||
1. For each VM navigate to the tab **VM Options** and click on **Edit Configuration**.
|
||||
2. Add the parameter `disk.EnableUUID` with a value of **TRUE**.
|
||||
|
||||
|
||||
|
||||
#### Using the GOVC CLI tool
|
||||
|
||||
You can also modify properties of VMs with the [govc](https://github.com/vmware/govmomi/tree/master/govc) command-line tool to enable disk UUIDs:
|
||||
|
||||
```sh
|
||||
$ govc vm.change -vm <vm-path> -e disk.enableUUID=TRUE
|
||||
```
|
||||
|
||||
#### Using Rancher node template
|
||||
|
||||
When creating new clusters in Rancher using vSphere node templates, you can configure the template to automatically enable disk UUIDs for all VMs created for a cluster:
|
||||
|
||||
1. Navigate to the **Node Templates** in the Rancher UI while logged in as admin user.
|
||||
|
||||
2. Add or edit an existing vSphere node template.
|
||||
|
||||
3. Under **Instance Options** click on **Add Parameter**.
|
||||
|
||||
4. Enter `disk.enableUUID` as key with a value of **TRUE**.
|
||||
|
||||
|
||||
|
||||
5. Click **Create** or **Save**.
|
||||
|
||||
### Troubleshooting
|
||||
|
||||
If you are experiencing issues while provisioning a cluster with enabled vSphere Cloud Provider or while creating vSphere volumes for your workloads, you should inspect the logs of the following K8s services:
|
||||
|
||||
- controller-manager (Manages volumes in vCenter)
|
||||
- kubelet: (Mounts vSphere volumes to pods)
|
||||
|
||||
If your cluster is not configured with external [Cluster Logging]({{< baseurl >}}/rancher/v2.x/en/tools/logging/), you will need to SSH into nodes to get the logs of the `kube-controller-manager` (running on one of the control plane nodes) and the `kubelet` (pertaining to the node where the stateful pod has been scheduled).
|
||||
|
||||
The easiest way to create a SSH session with a node is the Rancher CLI tool.
|
||||
|
||||
1. [Configure the Rancher CLI]({{< baseurl >}}/rancher/v2.x/en/cli/) for your cluster.
|
||||
2. Run the following command to get a shell to the corresponding nodes:
|
||||
|
||||
```sh
|
||||
$ rancher ssh <nodeName>
|
||||
```
|
||||
|
||||
3. Inspect the logs of the controller-manager and kubelet containers looking for errors related to the vSphere cloud provider:
|
||||
|
||||
```sh
|
||||
$ docker logs --since 15m kube-controller-manager
|
||||
$ docker logs --since 15m kubelet
|
||||
```
|
||||
|
||||
|
||||
### Related Links
|
||||
|
||||
- [vSphere Storage for Kubernetes](https://vmware.github.io/vsphere-storage-for-kubernetes/documentation/)
|
||||
- [Kubernetes Persistent Volumes](https://kubernetes.io/docs/concepts/storage/persistent-volumes/)
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 217 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 166 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 123 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 66 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 88 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 175 KiB |
Reference in New Issue
Block a user