From d66a3de7c0b73261e709a0dd2fbe691b93d65039 Mon Sep 17 00:00:00 2001 From: dkeightley Date: Sun, 24 May 2020 22:39:33 +1200 Subject: [PATCH] Updates for AWS cloudprovider with RKE - Update IAM role example (as per Rancher doc and [official](https://github.com/kubernetes/cloud-provider-aws) doc pages) - Update tagging wording --- .../cloud-providers/aws/_index.md | 158 ++++++++++++------ 1 file changed, 105 insertions(+), 53 deletions(-) diff --git a/content/rke/latest/en/config-options/cloud-providers/aws/_index.md b/content/rke/latest/en/config-options/cloud-providers/aws/_index.md index c580991232f..95df8b4be46 100644 --- a/content/rke/latest/en/config-options/cloud-providers/aws/_index.md +++ b/content/rke/latest/en/config-options/cloud-providers/aws/_index.md @@ -3,7 +3,7 @@ title: AWS Cloud Provider weight: 251 --- -To enable the AWS cloud provider, there are no RKE configuration options. You only need to set the name as `aws`. In order to use the AWS cloud provider, all cluster nodes must have already been configured with an [appropriate IAM role](#iam-requirements) and your AWS resources must be [tagged with a cluster ID](#tagging-amazon-resources). +To enable the AWS cloud provider, there are no RKE configuration options. You only need to set the name as `aws`. In order to use the AWS cloud provider, all cluster nodes must have already been configured with an [appropriate IAM role](#iam-requirements) and your AWS resources must be [tagged with a cluster ID](#tagging-aws-resources). ```yaml cloud_provider: @@ -12,19 +12,11 @@ cloud_provider: ## IAM Requirements -The nodes used in RKE that will be running the AWS cloud provider must have at least the following IAM policy (`rancher-role.json`). +In a cluster with the AWS cloud provider enabled, nodes must have at least the `ec2:Describe*` action. -```json -{ - "Effect": "Allow", - "Action": "ec2:Describe*", - "Resource": "*" -} -``` +In order to use Elastic Load Balancers (ELBs) and EBS volumes with Kubernetes, the node(s) will need to have the an IAM role with appropriate permissions. -In order to use Elastic Load Balancers (ELBs) and EBS with Kubernetes, the node(s) will need to have the an IAM role with appropriate access (`rancher-policy.json`). - -## Example Policy for IAM Role: +IAM policy for nodes with the `controlplane` role: ```json { @@ -32,64 +24,124 @@ In order to use Elastic Load Balancers (ELBs) and EBS with Kubernetes, the node( "Statement": [ { "Effect": "Allow", - "Action": "ec2:Describe*", - "Resource": "*" - }, - { - "Effect": "Allow", - "Action": "ec2:AttachVolume", - "Resource": "*" - }, - { - "Effect": "Allow", - "Action": "ec2:DetachVolume", - "Resource": "*" - }, - { - "Effect": "Allow", - "Action": ["elasticloadbalancing:*"], - "Resource": ["*"] + "Action": [ + "autoscaling:DescribeAutoScalingGroups", + "autoscaling:DescribeLaunchConfigurations", + "autoscaling:DescribeTags", + "ec2:DescribeInstances", + "ec2:DescribeRegions", + "ec2:DescribeRouteTables", + "ec2:DescribeSecurityGroups", + "ec2:DescribeSubnets", + "ec2:DescribeVolumes", + "ec2:CreateSecurityGroup", + "ec2:CreateTags", + "ec2:CreateVolume", + "ec2:ModifyInstanceAttribute", + "ec2:ModifyVolume", + "ec2:AttachVolume", + "ec2:AuthorizeSecurityGroupIngress", + "ec2:CreateRoute", + "ec2:DeleteRoute", + "ec2:DeleteSecurityGroup", + "ec2:DeleteVolume", + "ec2:DetachVolume", + "ec2:RevokeSecurityGroupIngress", + "ec2:DescribeVpcs", + "elasticloadbalancing:AddTags", + "elasticloadbalancing:AttachLoadBalancerToSubnets", + "elasticloadbalancing:ApplySecurityGroupsToLoadBalancer", + "elasticloadbalancing:CreateLoadBalancer", + "elasticloadbalancing:CreateLoadBalancerPolicy", + "elasticloadbalancing:CreateLoadBalancerListeners", + "elasticloadbalancing:ConfigureHealthCheck", + "elasticloadbalancing:DeleteLoadBalancer", + "elasticloadbalancing:DeleteLoadBalancerListeners", + "elasticloadbalancing:DescribeLoadBalancers", + "elasticloadbalancing:DescribeLoadBalancerAttributes", + "elasticloadbalancing:DetachLoadBalancerFromSubnets", + "elasticloadbalancing:DeregisterInstancesFromLoadBalancer", + "elasticloadbalancing:ModifyLoadBalancerAttributes", + "elasticloadbalancing:RegisterInstancesWithLoadBalancer", + "elasticloadbalancing:SetLoadBalancerPoliciesForBackendServer", + "elasticloadbalancing:AddTags", + "elasticloadbalancing:CreateListener", + "elasticloadbalancing:CreateTargetGroup", + "elasticloadbalancing:DeleteListener", + "elasticloadbalancing:DeleteTargetGroup", + "elasticloadbalancing:DescribeListeners", + "elasticloadbalancing:DescribeLoadBalancerPolicies", + "elasticloadbalancing:DescribeTargetGroups", + "elasticloadbalancing:DescribeTargetHealth", + "elasticloadbalancing:ModifyListener", + "elasticloadbalancing:ModifyTargetGroup", + "elasticloadbalancing:RegisterTargets", + "elasticloadbalancing:SetLoadBalancerPoliciesOfListener", + "iam:CreateServiceLinkedRole", + "kms:DescribeKey" + ], + "Resource": [ + "*" + ] } ] } ``` -Deploy files to AWS IAM: +IAM policy for nodes with the `etcd` or `worker` role: -```bash -$ aws iam create-instance-profile --instance-profile-name rancher-node -$ aws iam create-role --role-name rancher-node --assume-role-policy-document file://rancher-role.json -$ aws iam put-role-policy --role-name rancher-node --policy-name rancher-policy --policy-document file://rancher-policy.json -$ aws iam add-role-to-instance-profile --instance-profile rancher-node --role-name rancher-node +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Action": [ + "ec2:DescribeInstances", + "ec2:DescribeRegions", + "ecr:GetAuthorizationToken", + "ecr:BatchCheckLayerAvailability", + "ecr:GetDownloadUrlForLayer", + "ecr:GetRepositoryPolicy", + "ecr:DescribeRepositories", + "ecr:ListImages", + "ecr:BatchGetImage" + ], + "Resource": "*" + } + ] +} ``` -Set `IAM Instance Profile Name` in node template to `rancher-node` +## Tagging AWS Resources +The AWS cloud provider uses tagging to discover and manage resources, the following resources are not automatically tagged by Kubernetes or RKE: -## Tagging Amazon Resources +- **VPC**: The VPC used by the cluster +- **Subnet**: The subnets used by the cluster +- **EC2 instances**: All nodes launched for the cluster +- **Security Groups**: The security group(s) used by nodes in the cluster -Any resources used in a Kubernetes cluster with the Amazon cloud provider must be tagged with a cluster ID. + >**Note:** If creating a `LoadBalancer` service and there is more than one security group attached to nodes, you must tag only one of the security groups as `owned` so that Kubernetes knows which group to add and remove rules. A single untagged security group is allowed, however, sharing this between clusters is not recommended. -[Amazon Documentation: Tagging Your Amazon EC2 Resources](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_Tags.html) +[AWS Documentation: Tagging Your Amazon EC2 Resources](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_Tags.html) -The following resources need to tagged with a `ClusterID`: +You must tag with one of the following: -- **Nodes**: All hosts added in Rancher. -- **Subnet**: The subnet used for your cluster -- **Security Group**: The security group used for your cluster. +| Key | Value | +|---|---| +| kubernetes.io/cluster/`` | shared | - >**Note:** Do not tag multiple security groups. Tagging multiple groups generates an error when creating Elastic Load Balancer. +`` can be any string you choose. However, the same string must be used on every resource you tag. Setting the tag value to `owned` informs the cluster that all resources tagged with the `` are owned and managed by this cluster only. -The tag that should be used is: +If you do not share resources between clusters, you can change the tag to: -``` -Key=kubernetes.io/cluster/, Value=owned -``` +| Key | Value | +|---|---| +| kubernetes.io/cluster/`` | owned | -`` can be any string you choose. However, the same string must be used on every resource you tag. Setting the tag value to `owned` informs the cluster that all resources tagged with the `` are owned and managed by this cluster. +## Tagging for Load Balancers -If you share resources between clusters, you can change the tag to: +When provisioning a `LoadBalancer` service Kubernetes will attempt to discover the correct subnets, this is also achieved by tags and requires adding additional subnet tags to ensure internet-facing and internal ELBs are created in the correct subnets. -``` -Key=kubernetes.io/cluster/CLUSTERID, Value=shared -``` +[AWS Documentation: Subnet tagging for load balancers](https://docs.aws.amazon.com/eks/latest/userguide/load-balancing.html#subnet-tagging-for-load-balancers)