public/rancher-docs
1
0
Fork 0
mirror of https://github.com/rancher/rancher-docs.git synced 2026-09-26 04:58:07 +00:00
Code Issues Packages Projects Releases Wiki Activity

Remove intermediate /en folder

Browse Source
This commit is contained in:
Billy Tat
2022-08-17 09:14:41 -07:00
parent eb2197be30
commit 506e174643
441 changed files with 4 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/reference-guides/about-the-api/api-tokens.md
+66
View File
@@ -0,0 +1,66 @@
---
title: API Tokens
weight: 1
---
By default, some cluster-level API tokens are generated with infinite time-to-live (`ttl=0`). In other words, API tokens with `ttl=0` never expire unless you invalidate them. Tokens are not invalidated by changing a password.
You can deactivate API tokens by deleting them or by deactivating the user account.
### Deleting tokens
To delete a token,
1. Go to the list of all tokens in the Rancher API view at `https://<Rancher-Server-IP>/v3/tokens`.
1. Access the token you want to delete by its ID. For example, `https://<Rancher-Server-IP>/v3/tokens/kubectl-shell-user-vqkqt`
1. Click **Delete**.
Here is the complete list of tokens that are generated with `ttl=0`:
| Token | Description |
|-------|-------------|
| `kubeconfig-*` | Kubeconfig token |
| `kubectl-shell-*` | Access to `kubectl` shell in the browser |
| `agent-*` | Token for agent deployment |
| `compose-token-*` | Token for compose |
| `helm-token-*` | Token for Helm chart deployment |
| `*-pipeline*` | Pipeline token for project |
| `telemetry-*` | Telemetry token |
| `drain-node-*` | Token for drain (we use `kubectl` for drain because there is no native Kubernetes API) |
### Setting TTL on Kubeconfig Tokens
Admins can set a global TTL on Kubeconfig tokens. Once the token expires the kubectl command will require the user to authenticate to Rancher.
Go to the global settings and:
1. Set the `kubeconfig-generate-token` setting to `false`. This setting instructs Rancher to no longer automatically generate a token when a user clicks on download a kubeconfig file. The kubeconfig file will now provide a command to login to Rancher.
:::note
Once this setting is deactivated, a generated kubeconfig will reference the [Rancher CLI](../../pages-for-subheaders/cli-with-rancher.md) to retrieve a short lived token for the cluster. When you use this kubeconfig in a client, such as `kubectl`, the Rancher CLI needs to be installed as well.
:::
2. Set the `kubeconfig-token-ttl-minutes` setting to the desired duration in minutes. By default, `kubeconfig-token-ttl-minutes` is 960 (16 hours).
:::note
This value cannot exceed max-ttl of API tokens.(`https://<Rancher-Server-IP/v3/settings/auth-token-max-ttl-minutes`). `auth-token-max-ttl-minutes` is set to 1440 (24 hours) by default. `auth-token-max-ttl-minutes would default to 0 allowing tokens to never expire`.
:::
### Token Hashing
Users can enable token hashing, where tokens will undergo a one-way hash using the SHA256 algorithm. This is a non-reversible process, once enabled, this feature cannot be disabled. It is advisable to take backups prior to enabling and/or evaluated in a test environment first.
To enable token hashing, refer to [this section](../../pages-for-subheaders/enable-experimental-features.md).
This feature will affect all tokens which include, but are not limited to, the following:
- Kubeconfig tokens
- Bearer tokens API keys/calls
- Tokens used by internal operations
docs/reference-guides/amazon-eks-permissions/minimum-eks-permissions.md
+224
View File
@@ -0,0 +1,224 @@
---
title: Minimum EKS Permissions
weight: 1
---
Documented here is a minimum set of permissions necessary to use all functionality of the EKS driver in Rancher. Additional permissions are required for Rancher to provision the `Service Role` and `VPC` resources. Optionally these resources can be created **before** the cluster creation and will be selectable when defining the cluster configuration.
Resource | Description
---------|------------
Service Role | The service role provides Kubernetes the permissions it requires to manage resources on your behalf. Rancher can create the service role with the following [Service Role Permissions](#service-role-permissions).
VPC | Provides isolated network resources utilised by EKS and worker nodes. Rancher can create the VPC resources with the following [VPC Permissions](#vpc-permissions).
Resource targeting uses `*` as the ARN of many of the resources created cannot be known before creating the EKS cluster in Rancher.
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "EC2Permisssions",
"Effect": "Allow",
"Action": [
"ec2:RunInstances",
"ec2:RevokeSecurityGroupIngress",
"ec2:RevokeSecurityGroupEgress",
"ec2:DescribeInstanceTypes",
"ec2:DescribeRegions",
"ec2:DescribeVpcs",
"ec2:DescribeTags",
"ec2:DescribeSubnets",
"ec2:DescribeSecurityGroups",
"ec2:DescribeRouteTables",
"ec2:DescribeLaunchTemplateVersions",
"ec2:DescribeLaunchTemplates",
"ec2:DescribeKeyPairs",
"ec2:DescribeInternetGateways",
"ec2:DescribeImages",
"ec2:DescribeAvailabilityZones",
"ec2:DescribeAccountAttributes",
"ec2:DeleteTags",
"ec2:DeleteSecurityGroup",
"ec2:DeleteKeyPair",
"ec2:CreateTags",
"ec2:CreateSecurityGroup",
"ec2:CreateLaunchTemplateVersion",
"ec2:CreateLaunchTemplate",
"ec2:CreateKeyPair",
"ec2:AuthorizeSecurityGroupIngress",
"ec2:AuthorizeSecurityGroupEgress"
],
"Resource": "*"
},
{
"Sid": "CloudFormationPermisssions",
"Effect": "Allow",
"Action": [
"cloudformation:ListStacks",
"cloudformation:ListStackResources",
"cloudformation:DescribeStacks",
"cloudformation:DescribeStackResources",
"cloudformation:DescribeStackResource",
"cloudformation:DeleteStack",
"cloudformation:CreateStackSet",
"cloudformation:CreateStack"
],
"Resource": "*"
},
{
"Sid": "IAMPermissions",
"Effect": "Allow",
"Action": [
"iam:PassRole",
"iam:ListRoles",
"iam:ListRoleTags",
"iam:ListInstanceProfilesForRole",
"iam:ListInstanceProfiles",
"iam:ListAttachedRolePolicies",
"iam:GetRole",
"iam:GetInstanceProfile",
"iam:DetachRolePolicy",
"iam:DeleteRole",
"iam:CreateRole",
"iam:AttachRolePolicy"
],
"Resource": "*"
},
{
"Sid": "KMSPermisssions",
"Effect": "Allow",
"Action": "kms:ListKeys",
"Resource": "*"
},
{
"Sid": "EKSPermisssions",
"Effect": "Allow",
"Action": [
"eks:UpdateNodegroupVersion",
"eks:UpdateNodegroupConfig",
"eks:UpdateClusterVersion",
"eks:UpdateClusterConfig",
"eks:UntagResource",
"eks:TagResource",
"eks:ListUpdates",
"eks:ListTagsForResource",
"eks:ListNodegroups",
"eks:ListFargateProfiles",
"eks:ListClusters",
"eks:DescribeUpdate",
"eks:DescribeNodegroup",
"eks:DescribeFargateProfile",
"eks:DescribeCluster",
"eks:DeleteNodegroup",
"eks:DeleteFargateProfile",
"eks:DeleteCluster",
"eks:CreateNodegroup",
"eks:CreateFargateProfile",
"eks:CreateCluster"
],
"Resource": "*"
}
]
}
```
### Service Role Permissions
Permissions required for Rancher to create service role on users behalf during the EKS cluster creation process.
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "IAMPermisssions",
"Effect": "Allow",
"Action": [
"iam:AddRoleToInstanceProfile",
"iam:AttachRolePolicy",
"iam:CreateInstanceProfile",
"iam:CreateRole",
"iam:CreateServiceLinkedRole",
"iam:DeleteInstanceProfile",
"iam:DeleteRole",
"iam:DetachRolePolicy",
"iam:GetInstanceProfile",
"iam:GetRole",
"iam:ListAttachedRolePolicies",
"iam:ListInstanceProfiles",
"iam:ListInstanceProfilesForRole",
"iam:ListRoles",
"iam:ListRoleTags",
"iam:PassRole",
"iam:RemoveRoleFromInstanceProfile"
],
"Resource": "*"
}
]
}
```
When an EKS cluster is created, Rancher will create a service role with the following trust policy:
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Action": "sts:AssumeRole",
"Principal": {
"Service": "eks.amazonaws.com"
},
"Effect": "Allow",
"Sid": ""
}
]
}
```
This role will also have two role policy attachments with the following policies ARNs:
```
arn:aws:iam::aws:policy/AmazonEKSClusterPolicy
arn:aws:iam::aws:policy/AmazonEKSServicePolicy
```
### VPC Permissions
Permissions required for Rancher to create VPC and associated resources.
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VPCPermissions",
"Effect": "Allow",
"Action": [
"ec2:ReplaceRoute",
"ec2:ModifyVpcAttribute",
"ec2:ModifySubnetAttribute",
"ec2:DisassociateRouteTable",
"ec2:DetachInternetGateway",
"ec2:DescribeVpcs",
"ec2:DeleteVpc",
"ec2:DeleteTags",
"ec2:DeleteSubnet",
"ec2:DeleteRouteTable",
"ec2:DeleteRoute",
"ec2:DeleteInternetGateway",
"ec2:CreateVpc",
"ec2:CreateSubnet",
"ec2:CreateSecurityGroup",
"ec2:CreateRouteTable",
"ec2:CreateRoute",
"ec2:CreateInternetGateway",
"ec2:AttachInternetGateway",
"ec2:AssociateRouteTable"
],
"Resource": "*"
}
]
}
```
docs/reference-guides/backup-restore-configuration/backup-configuration.md
+180
View File
@@ -0,0 +1,180 @@
---
title: Backup Configuration
shortTitle: Backup
weight: 1
---
The Backup Create page lets you configure a schedule, enable encryption and specify the storage location for your backups.
- [Schedule](#schedule)
- [Encryption](#encryption)
- [Storage Location](#storage-location)
- [S3](#s3)
- [Example S3 Storage Configuration](#example-s3-storage-configuration)
- [Example MinIO Configuration](#example-minio-configuration)
- [Example credentialSecret](#example-credentialsecret)
- [IAM Permissions for EC2 Nodes to Access S3](#iam-permissions-for-ec2-nodes-to-access-s3)
- [Examples](#examples)
# Schedule
Select the first option to perform a one-time backup, or select the second option to schedule recurring backups. Selecting **Recurring Backups** lets you configure following two fields:
- **Schedule**: This field accepts
- Standard [cron expressions](https://en.wikipedia.org/wiki/Cron), such as `"0 * * * *"`
- Descriptors, such as `"@midnight"` or `"@every 1h30m"`
- **Retention Count**: This value specifies how many backup files must be retained. If files exceed the given retentionCount, the oldest files will be deleted. The default value is 10.
| YAML Directive Name | Description |
| ---------------- | ---------------- |
| `schedule` | Provide the cron string for scheduling recurring backups. |
| `retentionCount` | Provide the number of backup files to be retained. |
# Encryption
The rancher-backup gathers resources by making calls to the kube-apiserver. Objects returned by apiserver are decrypted, so even if [encryption At rest](https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/) is enabled, even the encrypted objects gathered by the backup will be in plaintext.
To avoid storing them in plaintext, you can use the same encryptionConfig file that was used for at-rest encryption, to encrypt certain resources in your backup.
:::note Important:
You must save the encryptionConfig file, because it won’t be saved by the rancher-backup operator.
The same encryptionFile needs to be used when performing a restore.
:::
The operator consumes this encryptionConfig as a Kubernetes Secret, and the Secret must be in the operator’s namespace. Rancher installs the `rancher-backup` operator in the `cattle-resources-system` namespace, so create this encryptionConfig secret in that namespace.
For the `EncryptionConfiguration`, you can use the [sample file provided in the Kubernetes documentation.](https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/#understanding-the-encryption-at-rest-configuration)
To create the Secret, the encryption configuration file must be named `encryption-provider-config.yaml`, and the `--from-file` flag must be used to create this secret.
Save the `EncryptionConfiguration` in a file called `encryption-provider-config.yaml` and run this command:
```
kubectl create secret generic encryptionconfig \
--from-file=./encryption-provider-config.yaml \
-n cattle-resources-system
```
This will ensure that the secret contains a key named `encryption-provider-config.yaml`, and the operator will use this key to get the encryption configuration.
The `Encryption Config Secret` dropdown will filter out and list only those Secrets that have this exact key
![](/img/backup_restore/backup/encryption.png)
In the example command above, the name `encryptionconfig` can be changed to anything.
| YAML Directive Name | Description |
| ---------------- | ---------------- |
| `encryptionConfigSecretName` | Provide the name of the Secret from `cattle-resources-system` namespace, that contains the encryption config file. |
# Storage Location
If the StorageLocation is specified in the Backup, the operator will retrieve the backup location from that particular S3 bucket. If not specified, the operator will try to find this file in the default operator-level S3 store, and in the operator-level PVC store. The default storage location is configured during the deployment of the `rancher-backup` operator.
Selecting the first option stores this backup in the storage location configured while installing the rancher-backup chart. The second option lets you configure a different S3 compatible storage provider for storing the backup.
### S3
The S3 storage location contains the following configuration fields:
1. **Credential Secret** (optional): If you need to use the AWS Access keys Secret keys to access s3 bucket, create a secret with your credentials with keys and the directives `accessKey` and `secretKey`. It can be in any namespace. An example secret is [here.](#example-credentialsecret) This directive is unnecessary if the nodes running your operator are in EC2 and set up with IAM permissions that allow them to access S3, as described in [this section.](#iam-permissions-for-ec2-nodes-to-access-s3) The Credential Secret dropdown lists the secrets in all namespaces.
1. **Bucket Name**: The name of the S3 bucket where backup files will be stored.
1. **Region** (optional): The AWS [region](https://aws.amazon.com/about-aws/global-infrastructure/regions_az/) where the S3 bucket is located. This field isn't needed for configuring MinIO.
1. **Folder** (optional): The name of the folder in the S3 bucket where backup files will be stored. Nested folders (e.g., `rancher/cluster1`) are not supported.
1. **Endpoint**: The [endpoint](https://docs.aws.amazon.com/general/latest/gr/s3.html) that is used to access S3 in the region of your bucket.
1. **Endpoint CA** (optional): This should be the Base64 encoded CA cert. For an example, refer to the [example S3 compatible configuration.](#example-s3-storage-configuration)
1. **Skip TLS Verifications** (optional): Set to true if you are not using TLS.
| YAML Directive Name | Description | Required |
| ---------------- | ---------------- | ------------ |
| `credentialSecretName` | If you need to use the AWS Access keys Secret keys to access s3 bucket, create a secret with your credentials with keys and the directives `accessKey` and `secretKey`. It can be in any namespace as long as you provide that namespace in `credentialSecretNamespace`. An example secret is [here.](#example-credentialsecret) This directive is unnecessary if the nodes running your operator are in EC2 and set up with IAM permissions that allow them to access S3, as described in [this section.](#iam-permissions-for-ec2-nodes-to-access-s3) | |
| `credentialSecretNamespace` | The namespace of the secret containing the credentials to access S3. This directive is unnecessary if the nodes running your operator are in EC2 and set up with IAM permissions that allow them to access S3, as described in [this section.](#iam-permissions-for-ec2-nodes-to-access-s3) | |
| `bucketName` | The name of the S3 bucket where backup files will be stored. | ✓ |
| `folder` | The name of the folder in the S3 bucket where backup files will be stored. Nested folders (e.g., `rancher/cluster1`) are not supported. | |
| `region` | The AWS [region](https://aws.amazon.com/about-aws/global-infrastructure/regions_az/) where the S3 bucket is located. | ✓ |
| `endpoint` | The [endpoint](https://docs.aws.amazon.com/general/latest/gr/s3.html) that is used to access S3 in the region of your bucket. | ✓ |
| `endpointCA` | This should be the Base64 encoded CA cert. For an example, refer to the [example S3 compatible configuration.](#example-s3-storage-configuration) | |
| `insecureTLSSkipVerify` | Set to true if you are not using TLS. | |
### Example S3 Storage Configuration
```yaml
s3:
credentialSecretName: s3-creds
credentialSecretNamespace: default
bucketName: rancher-backups
folder: rancher
region: us-west-2
endpoint: s3.us-west-2.amazonaws.com
```
### Example MinIO Configuration
```yaml
s3:
credentialSecretName: minio-creds
bucketName: rancherbackups
endpoint: minio.35.202.130.254.xip.io
endpointCA: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURHakNDQWdLZ0F3SUJBZ0lKQUtpWFZpNEpBb0J5TUEwR0NTcUdTSWIzRFFFQkN3VUFNQkl4RURBT0JnTlYKQkFNTUIzUmxjM1F0WTJFd0hoY05NakF3T0RNd01UZ3lOVFE1V2hjTk1qQXhNREk1TVRneU5UUTVXakFTTVJBdwpEZ1lEVlFRRERBZDBaWE4wTFdOaE1JSUJJakFOQmdrcWhraUc5dzBCQVFFRkFBT0NBUThBTUlJQkNnS0NBUUVBCjA4dnV3Q2Y0SEhtR2Q2azVNTmozRW5NOG00T2RpS3czSGszd1NlOUlXQkwyVzY5WDZxenBhN2I2M3U2L05mMnkKSnZWNDVqeXplRFB6bFJycjlpbEpWaVZ1NFNqWlFjdG9jWmFCaVNsL0xDbEFDdkFaUlYvKzN0TFVTZSs1ZDY0QQpWcUhDQlZObU5xM3E3aVY0TE1aSVpRc3N6K0FxaU1Sd0pOMVVKQTZ6V0tUc2Yzc3ByQ0J2dWxJWmZsVXVETVAyCnRCTCt6cXZEc0pDdWlhNEEvU2JNT29tVmM2WnNtTGkwMjdub3dGRld3MnRpSkM5d0xMRE14NnJoVHQ4a3VvVHYKQXJpUjB4WktiRU45L1Uzb011eUVKbHZyck9YS2ZuUDUwbk8ycGNaQnZCb3pUTStYZnRvQ1d5UnhKUmI5cFNTRApKQjlmUEFtLzNZcFpMMGRKY2sxR1h3SURBUUFCbzNNd2NUQWRCZ05WSFE0RUZnUVU5NHU4WXlMdmE2MTJnT1pyCm44QnlFQ2NucVFjd1FnWURWUjBqQkRzd09ZQVU5NHU4WXlMdmE2MTJnT1pybjhCeUVDY25xUWVoRnFRVU1CSXgKRURBT0JnTlZCQU1NQjNSbGMzUXRZMkdDQ1FDb2wxWXVDUUtBY2pBTUJnTlZIUk1FQlRBREFRSC9NQTBHQ1NxRwpTSWIzRFFFQkN3VUFBNElCQVFER1JRZ1RtdzdVNXRQRHA5Q2psOXlLRW9Vd2pYWWM2UlAwdm1GSHpubXJ3dUVLCjFrTkVJNzhBTUw1MEpuS29CY0ljVDNEeGQ3TGdIbTNCRE5mVVh2anArNnZqaXhJYXR2UWhsSFNVaWIyZjJsSTkKVEMxNzVyNCtROFkzelc1RlFXSDdLK08vY3pJTGh5ei93aHRDUlFkQ29lS1dXZkFiby8wd0VSejZzNkhkVFJzNwpHcWlGNWZtWGp6S0lOcTBjMHRyZ0xtalNKd1hwSnU0ZnNGOEcyZUh4b2pOKzdJQ1FuSkg5cGRIRVpUQUtOL2ppCnIvem04RlZtd1kvdTBndEZneWVQY1ZWbXBqRm03Y0ZOSkc4Y2ZYd0QzcEFwVjhVOGNocTZGeFBHTkVvWFZnclMKY1VRMklaU0RJd1FFY3FvSzFKSGdCUWw2RXBaUVpWMW1DRklrdFBwSQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0t
```
### Example credentialSecret
```yaml
apiVersion: v1
kind: Secret
metadata:
name: creds
type: Opaque
data:
accessKey: <Enter your base64-encoded access key>
secretKey: <Enter your base64-encoded secret key>
```
### IAM Permissions for EC2 Nodes to Access S3
There are two ways to set up the `rancher-backup` operator to use S3 as the backup storage location.
One way is to configure the `credentialSecretName` in the Backup custom resource, which refers to AWS credentials that have access to S3.
If the cluster nodes are in Amazon EC2, the S3 access can also be set up by assigning IAM permissions to the EC2 nodes so that they can access S3.
To allow a node to access S3, follow the instructions in the [AWS documentation](https://aws.amazon.com/premiumsupport/knowledge-center/ec2-instance-access-s3-bucket/) to create an IAM role for EC2. When you add a custom policy to the role, add the following permissions, and replace the `Resource` with your bucket name:
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::rancher-backups"
]
},
{
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:GetObject",
"s3:DeleteObject",
"s3:PutObjectAcl"
],
"Resource": [
"arn:aws:s3:::rancher-backups/*"
]
}
]
}
```
After the role is created, and you have attached the corresponding instance profile to your EC2 instance(s), the `credentialSecretName` directive can be left empty in the Backup custom resource.
# Examples
For example Backup custom resources, refer to [this page.](examples.md#backup)
docs/reference-guides/backup-restore-configuration/examples.md
+308
View File
@@ -0,0 +1,308 @@
---
title: Examples
weight: 5
---
This section contains examples of Backup and Restore custom resources.
The default backup storage location is configured when the `rancher-backup` operator is installed or upgraded.
Encrypted backups can only be restored if the Restore custom resource uses the same encryption configuration secret that was used to create the backup.
- [Backup](#backup)
- [Backup in the default location with encryption](#backup-in-the-default-location-with-encryption)
- [Recurring backup in the default location](#recurring-backup-in-the-default-location)
- [Encrypted recurring backup in the default location](#encrypted-recurring-backup-in-the-default-location)
- [Encrypted backup in Minio](#encrypted-backup-in-minio)
- [Backup in S3 using AWS credential secret](#backup-in-s3-using-aws-credential-secret)
- [Recurring backup in S3 using AWS credential secret](#recurring-backup-in-s3-using-aws-credential-secret)
- [Backup from EC2 nodes with IAM permission to access S3](#backup-from-ec2-nodes-with-iam-permission-to-access-s3)
- [Restore](#restore)
- [Restore using the default backup file location](#restore-using-the-default-backup-file-location)
- [Restore for Rancher migration](#restore-for-rancher-migration)
- [Restore from encrypted backup](#restore-from-encrypted-backup)
- [Restore an encrypted backup from Minio](#restore-an-encrypted-backup-from-minio)
- [Restore from backup using an AWS credential secret to access S3](#restore-from-backup-using-an-aws-credential-secret-to-access-s3)
- [Restore from EC2 nodes with IAM permissions to access S3](#restore-from-ec2-nodes-with-iam-permissions-to-access-s3)
- [Example Credential Secret for Storing Backups in S3](#example-credential-secret-for-storing-backups-in-s3)
- [Example EncryptionConfiguration](#example-encryptionconfiguration)
# Backup
This section contains example Backup custom resources.
### Backup in the Default Location with Encryption
```yaml
apiVersion: resources.cattle.io/v1
kind: Backup
metadata:
name: default-location-encrypted-backup
spec:
resourceSetName: rancher-resource-set
encryptionConfigSecretName: encryptionconfig
```
### Recurring Backup in the Default Location
```yaml
apiVersion: resources.cattle.io/v1
kind: Backup
metadata:
name: default-location-recurring-backup
spec:
resourceSetName: rancher-resource-set
schedule: "@every 1h"
retentionCount: 10
```
### Encrypted Recurring Backup in the Default Location
```yaml
apiVersion: resources.cattle.io/v1
kind: Backup
metadata:
name: default-enc-recurring-backup
spec:
resourceSetName: rancher-resource-set
encryptionConfigSecretName: encryptionconfig
schedule: "@every 1h"
retentionCount: 3
```
### Encrypted Backup in Minio
```yaml
apiVersion: resources.cattle.io/v1
kind: Backup
metadata:
name: minio-backup
spec:
storageLocation:
s3:
credentialSecretName: minio-creds
credentialSecretNamespace: default
bucketName: rancherbackups
endpoint: minio.xip.io
endpointCA: LS0tLS1CRUdJTi3VUFNQkl5UUT.....pbEpWaVzNkRS0tLS0t
resourceSetName: rancher-resource-set
encryptionConfigSecretName: encryptionconfig
```
### Backup in S3 Using AWS Credential Secret
```yaml
apiVersion: resources.cattle.io/v1
kind: Backup
metadata:
name: s3-backup
spec:
storageLocation:
s3:
credentialSecretName: s3-creds
credentialSecretNamespace: default
bucketName: rancher-backups
folder: ecm1
region: us-west-2
endpoint: s3.us-west-2.amazonaws.com
resourceSetName: rancher-resource-set
encryptionConfigSecretName: encryptionconfig
```
### Recurring Backup in S3 Using AWS Credential Secret
```yaml
apiVersion: resources.cattle.io/v1
kind: Backup
metadata:
name: s3-recurring-backup
spec:
storageLocation:
s3:
credentialSecretName: s3-creds
credentialSecretNamespace: default
bucketName: rancher-backups
folder: ecm1
region: us-west-2
endpoint: s3.us-west-2.amazonaws.com
resourceSetName: rancher-resource-set
encryptionConfigSecretName: encryptionconfig
schedule: "@every 1h"
retentionCount: 10
```
### Backup from EC2 Nodes with IAM Permission to Access S3
This example shows that the AWS credential secret does not have to be provided to create a backup if the nodes running `rancher-backup` have [these permissions for access to S3.](backup-configuration.md#iam-permissions-for-ec2-nodes-to-access-s3)
```yaml
apiVersion: resources.cattle.io/v1
kind: Backup
metadata:
name: s3-iam-backup
spec:
storageLocation:
s3:
bucketName: rancher-backups
folder: ecm1
region: us-west-2
endpoint: s3.us-west-2.amazonaws.com
resourceSetName: rancher-resource-set
encryptionConfigSecretName: encryptionconfig
```
# Restore
This section contains example Restore custom resources.
### Restore Using the Default Backup File Location
```yaml
apiVersion: resources.cattle.io/v1
kind: Restore
metadata:
name: restore-default
spec:
backupFilename: default-location-recurring-backup-752ecd87-d958-4d20-8350-072f8d090045-2020-09-26T12-29-54-07-00.tar.gz
# encryptionConfigSecretName: test-encryptionconfig
```
### Restore for Rancher Migration
```yaml
apiVersion: resources.cattle.io/v1
kind: Restore
metadata:
name: restore-migration
spec:
backupFilename: backup-b0450532-cee1-4aa1-a881-f5f48a007b1c-2020-09-15T07-27-09Z.tar.gz
prune: false
storageLocation:
s3:
credentialSecretName: s3-creds
credentialSecretNamespace: default
bucketName: rancher-backups
folder: ecm1
region: us-west-2
endpoint: s3.us-west-2.amazonaws.com
```
### Restore from Encrypted Backup
```yaml
apiVersion: resources.cattle.io/v1
kind: Restore
metadata:
name: restore-encrypted
spec:
backupFilename: default-test-s3-def-backup-c583d8f2-6daf-4648-8ead-ed826c591471-2020-08-24T20-47-05Z.tar.gz
encryptionConfigSecretName: encryptionconfig
```
### Restore an Encrypted Backup from Minio
```yaml
apiVersion: resources.cattle.io/v1
kind: Restore
metadata:
name: restore-minio
spec:
backupFilename: default-minio-backup-demo-aa5c04b7-4dba-4c48-9ac4-ab7916812eaa-2020-08-30T13-18-17-07-00.tar.gz
storageLocation:
s3:
credentialSecretName: minio-creds
credentialSecretNamespace: default
bucketName: rancherbackups
endpoint: minio.xip.io
endpointCA: LS0tLS1CRUdJTi3VUFNQkl5UUT.....pbEpWaVzNkRS0tLS0t
encryptionConfigSecretName: test-encryptionconfig
```
### Restore from Backup Using an AWS Credential Secret to Access S3
```yaml
apiVersion: resources.cattle.io/v1
kind: Restore
metadata:
name: restore-s3-demo
spec:
backupFilename: test-s3-recurring-backup-752ecd87-d958-4d20-8350-072f8d090045-2020-09-26T12-49-34-07-00.tar.gz.enc
storageLocation:
s3:
credentialSecretName: s3-creds
credentialSecretNamespace: default
bucketName: rancher-backups
folder: ecm1
region: us-west-2
endpoint: s3.us-west-2.amazonaws.com
encryptionConfigSecretName: test-encryptionconfig
```
### Restore from EC2 Nodes with IAM Permissions to Access S3
This example shows that the AWS credential secret does not have to be provided to restore from backup if the nodes running `rancher-backup` have [these permissions for access to S3.](backup-configuration.md#iam-permissions-for-ec2-nodes-to-access-s3)
```yaml
apiVersion: resources.cattle.io/v1
kind: Restore
metadata:
name: restore-s3-demo
spec:
backupFilename: default-test-s3-recurring-backup-84bf8dd8-0ef3-4240-8ad1-fc7ec308e216-2020-08-24T10#52#44-07#00.tar.gz
storageLocation:
s3:
bucketName: rajashree-backup-test
folder: ecm1
region: us-west-2
endpoint: s3.us-west-2.amazonaws.com
encryptionConfigSecretName: test-encryptionconfig
```
# Example Credential Secret for Storing Backups in S3
```yaml
apiVersion: v1
kind: Secret
metadata:
name: creds
type: Opaque
data:
accessKey: <Enter your base64-encoded access key>
secretKey: <Enter your base64-encoded secret key>
```
# Example EncryptionConfiguration
The snippet below demonstrates two different types of secrets and their relevance with respect to Backup and Restore of custom resources.
The first example is that of a secret that is used to encrypt the backup files. The backup operator, in this case, will not be able to read the secrets encryption file. It only uses the contents of the secret.
The second example is that of a Kubernetes secrets encryption config file that is used to encrypt secrets when stored in etcd. **When backing up the etcd datastore, be sure to also back up the EncryptionConfiguration.** Failure to do so will result in an inability to use the restored data if secrets encryption was in use at the time the data was backed up.
```yaml
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
- secrets
providers:
- aesgcm:
keys:
- name: key1
secret: c2VjcmV0IGlzIHNlY3VyZQ==
- name: key2
secret: dGhpcyBpcyBwYXNzd29yZA==
- aescbc:
keys:
- name: key1
secret: c2VjcmV0IGlzIHNlY3VyZQ==
- name: key2
secret: dGhpcyBpcyBwYXNzd29yZA==
- secretbox:
keys:
- name: key1
secret: YWJjZGVmZ2hpamtsbW5vcHFyc3R1dnd4eXoxMjM0NTY=
```
docs/reference-guides/backup-restore-configuration/restore-configuration.md
+90
View File
@@ -0,0 +1,90 @@
---
title: Restore Configuration
shortTitle: Restore
weight: 2
---
The Restore Create page lets you provide details of the backup to restore from
![](/img/backup_restore/restore/restore.png)
- [Backup Source](#backup-source)
- [An Existing Backup Config](#an-existing-backup-config)
- [The default storage target](#the-default-storage-target)
- [An S3-compatible object store](#an-s3-compatible-object-store)
- [Encryption](#encryption)
- [Prune during restore](#prune-during-restore)
- [Getting the Backup Filename from S3](#getting-the-backup-filename-from-s3)
# Backup Source
Provide details of the backup file and its storage location, which the operator will then use to perform the restore. Select from the following options to provide these details
### An existing backup config
Selecting this option will populate the **Target Backup** dropdown with the Backups available in this cluster. Select the Backup from the dropdown, and that will fill out the **Backup Filename** field for you, and will also pass the backup source information from the selected Backup to the operator.
![](/img/backup_restore/restore/existing.png)
If the Backup custom resource does not exist in the cluster, you need to get the exact filename and provide the backup source details with the default storage target or an S3-compatible object store.
### The default storage target
Select this option if you are restoring from a backup file that exists in the default storage location configured at the operator-level. The operator-level configuration is the storage location that was configured when the `rancher-backup` operator was installed or upgraded. Provide the exact filename in the **Backup Filename** field.
![](/img/backup_restore/restore/default.png)
### An S3-compatible object store
Select this option if no default storage location is configured at the operator-level, OR if the backup file exists in a different S3 bucket than the one configured as the default storage location. Provide the exact filename in the **Backup Filename** field. Refer to [this section](#getting-the-backup-filename-from-s3) for exact steps on getting the backup filename from s3. Fill in all the details for the S3 compatible object store. Its fields are exactly same as ones for the `backup.StorageLocation` configuration in the [Backup custom resource.](backup-configuration.md#storage-location)
![](/img/backup_restore/restore/s3store.png)
# Encryption
If the backup was created with encryption enabled, its file will have `.enc` suffix. Choosing such a Backup, or providing a backup filename with `.enc` suffix will display another dropdown named **Encryption Config Secret**.
![](/img/backup_restore/restore/encryption.png)
The Secret selected from this dropdown must have the same contents as the one used for the Backup custom resource while performing the backup. If the encryption configuration doesn't match, the restore will fail
The `Encryption Config Secret` dropdown will filter out and list only those Secrets that have this exact key
| YAML Directive Name | Description |
| ---------------- | ---------------- |
| `encryptionConfigSecretName` | Provide the name of the Secret from `cattle-resources-system` namespace, that contains the encryption config file. |
:::note Important:
This field should only be set if the backup was created with encryption enabled. Providing the incorrect encryption config will cause the restore to fail.
:::
# Prune During Restore
* **Prune**: In order to fully restore Rancher from a backup, and to go back to the exact state it was at when the backup was performed, we need to delete any additional resources that were created by Rancher after the backup was taken. The operator does so if the **Prune** flag is enabled. Prune is enabled by default and it is recommended to keep it enabled.
* **Delete Timeout**: This is the amount of time the operator will wait while deleting a resource before editing the resource to remove finalizers and attempt deletion again.
| YAML Directive Name | Description |
| ---------------- | ---------------- |
| `prune` | Delete the resources managed by Rancher that are not present in the backup (Recommended). |
| `deleteTimeoutSeconds` | Amount of time the operator will wait while deleting a resource before editing the resource to remove finalizers and attempt deletion again. |
# Getting the Backup Filename from S3
This is the name of the backup file that the `rancher-backup` operator will use to perform the restore.
To obtain this file name from S3, go to your S3 bucket (and folder if it was specified while performing backup).
Copy the filename and store it in your Restore custom resource. So assuming the name of your backup file is `backupfile`,
- If your bucket name is `s3bucket` and no folder was specified, then the `backupFilename` to use will be `backupfile`.
- If your bucket name is `s3bucket` and the base folder is`s3folder`, the `backupFilename` to use is only `backupfile` .
- If there is a subfolder inside `s3Folder` called `s3sub`, and that has your backup file, then the `backupFilename` to use is `s3sub/backupfile`.
| YAML Directive Name | Description |
| ---------------- | ---------------- |
| `backupFilename` | This is the name of the backup file that the `rancher-backup` operator will use to perform the restore. |
docs/reference-guides/backup-restore-configuration/storage-configuration.md
+65
View File
@@ -0,0 +1,65 @@
---
title: Backup Storage Location Configuration
shortTitle: Storage
weight: 3
---
Configure a storage location where all backups are saved by default. You will have the option to override this with each backup, but will be limited to using an S3-compatible object store.
Only one storage location can be configured at the operator level.
- [Storage Location Configuration](#storage-location-configuration)
- [No Default Storage Location](#no-default-storage-location)
- [S3-compatible Object Store](#s3-compatible-object-store)
- [Use an existing StorageClass](#existing-storageclass)
- [Use an existing PersistentVolume](#existing-persistent-volume)
- [Encryption](#encryption)
- [Example values.yaml for the rancher-backup Helm Chart](#example-values-yaml-for-the-rancher-backup-helm-chart)
# Storage Location Configuration
### No Default Storage Location
You can choose to not have any operator-level storage location configured. If you select this option, you must configure an S3-compatible object store as the storage location for each individual backup.
### S3-compatible Object Store
| Parameter | Description |
| -------------- | -------------- |
| Credential Secret | Choose the credentials for S3 from your secrets in Rancher. [Example](examples.md#example-credential-secret-for-storing-backups-in-s3). |
| Bucket Name | Enter the name of the [S3 bucket](https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html) where the backups will be stored. Default: `rancherbackups`. |
| Region | The [AWS region](https://aws.amazon.com/about-aws/global-infrastructure/regions_az/) where the S3 bucket is located. |
| Folder | The [folder in the S3 bucket](https://docs.aws.amazon.com/AmazonS3/latest/user-guide/using-folders.html) where the backups will be stored. |
| Endpoint | The [S3 endpoint](https://docs.aws.amazon.com/general/latest/gr/s3.html) For example, `s3.us-west-2.amazonaws.com`. |
| Endpoint CA | The CA cert used to for the S3 endpoint. Default: base64 encoded CA cert |
| insecureTLSSkipVerify | Set to true if you are not using TLS. |
### Existing StorageClass
Installing the `rancher-backup` chart by selecting the StorageClass option will create a Persistent Volume Claim (PVC), and Kubernetes will in turn dynamically provision a Persistent Volume (PV) where all the backups will be saved by default.
For information about creating storage classes refer to [this section.](../../how-to-guides/advanced-user-guides/manage-clusters/create-kubernetes-persistent-storage/manage-persistent-storage/dynamically-provision-new-storage.md)
::: note Important:
It is highly recommended to use a StorageClass with a reclaim policy of "Retain". Otherwise if the PVC created by the `rancher-backup` chart gets deleted (either during app upgrade, or accidentally), the PV will get deleted too, which means all backups saved in it will get deleted.
If no such StorageClass is available, after the PV is provisioned, make sure to edit its reclaim policy and set it to "Retain" before storing backups in it.
:::
### Existing Persistent Volume
Select an existing Persistent Volume (PV) that will be used to store your backups. For information about creating PersistentVolumes in Rancher, refer to [this section.](../../how-to-guides/advanced-user-guides/manage-clusters/create-kubernetes-persistent-storage/manage-persistent-storage/set-up-existing-storage.md#2-add-a-persistent-volume-that-refers-to-the-persistent-storage)
:::note Important:
It is highly recommended to use a Persistent Volume with a reclaim policy of "Retain". Otherwise if the PVC created by the `rancher-backup` chart gets deleted (either during app upgrade, or accidentally), the PV will get deleted too, which means all backups saved in it will get deleted.
:::
# Example values.yaml for the rancher-backup Helm Chart
The documented `values.yaml` file that can be used to configure `rancher-backup` operator when the Helm CLI is used can be found in the [backup-restore-operator repository.](https://github.com/rancher/backup-restore-operator/blob/master/charts/rancher-backup/values.yaml)
For more information about `values.yaml` files and configuring Helm charts during installation, refer to the [Helm documentation.](https://helm.sh/docs/intro/using_helm/#customizing-the-chart-before-installing)
docs/reference-guides/best-practices/rancher-managed-clusters/logging-best-practices.md
+88
View File
@@ -0,0 +1,88 @@
---
title: Logging Best Practices
weight: 1
---
In this guide, we recommend best practices for cluster-level logging and application logging.
- [Cluster-level Logging](#cluster-level-logging)
- [Application Logging](#application-logging)
- [General Best Practices](#general-best-practices)
Before Rancher v2.5, logging in Rancher has historically been a pretty static integration. There were a fixed list of aggregators to choose from (ElasticSearch, Splunk, Kafka, Fluentd and Syslog), and only two configuration points to choose (Cluster-level and Project-level).
Rancher provides a flexible experience for log aggregation. With the logging feature, administrators and users alike can deploy logging that meets fine-grained collection criteria while offering a wider array of destinations and configuration options.
"Under the hood", Rancher logging uses the Banzai Cloud logging operator. We provide manageability of this operator (and its resources), and tie that experience in with managing your Rancher clusters.
# Cluster-level Logging
### Cluster-wide Scraping
For some users, it is desirable to scrape logs from every container running in the cluster. This usually coincides with your security team's request (or requirement) to collect all logs from all points of execution.
In this scenario, it is recommended to create at least two _ClusterOutput_ objects - one for your security team (if you have that requirement), and one for yourselves, the cluster administrators. When creating these objects take care to choose an output endpoint that can handle the significant log traffic coming from the entire cluster. Also make sure to choose an appropriate index to receive all these logs.
Once you have created these _ClusterOutput_ objects, create a _ClusterFlow_ to collect all the logs. Do not define any _Include_ or _Exclude_ rules on this flow. This will ensure that all logs from across the cluster are collected. If you have two _ClusterOutputs_, make sure to send logs to both of them.
### Kubernetes Components
_ClusterFlows_ have the ability to collect logs from all containers on all hosts in the Kubernetes cluster. This works well in cases where those containers are part of a Kubernetes pod; however, RKE containers exist outside of the scope of Kubernetes.
Currently the logs from RKE containers are collected, but are not able to easily be filtered. This is because those logs do not contain information as to the source container (e.g. `etcd` or `kube-apiserver`).
A future release of Rancher will include the source container name which will enable filtering of these component logs. Once that change is made, you will be able to customize a _ClusterFlow_ to retrieve **only** the Kubernetes component logs, and direct them to an appropriate output.
# Application Logging
Best practice not only in Kubernetes but in all container-based applications is to direct application logs to `stdout`/`stderr`. The container runtime will then trap these logs and do **something** with them - typically writing them to a file. Depending on the container runtime (and its configuration), these logs can end up in any number of locations.
In the case of writing the logs to a file, Kubernetes helps by creating a `/var/log/containers` directory on each host. This directory symlinks the log files to their actual destination (which can differ based on configuration or container runtime).
Rancher logging will read all log entries in `/var/log/containers`, ensuring that all log entries from all containers (assuming a default configuration) will have the opportunity to be collected and processed.
### Specific Log Files
Log collection only retrieves `stdout`/`stderr` logs from pods in Kubernetes. But what if we want to collect logs from other files that are generated by applications? Here, a log streaming sidecar (or two) may come in handy.
The goal of setting up a streaming sidecar is to take log files that are written to disk, and have their contents streamed to `stdout`. This way, the Banzai Logging Operator can pick up those logs and send them to your desired output.
To set this up, edit your workload resource (e.g. Deployment) and add the following sidecar definition:
```
...
containers:
- args:
- -F
- /path/to/your/log/file.log
command:
- tail
image: busybox
name: stream-log-file-[name]
volumeMounts:
- mountPath: /path/to/your/log
name: mounted-log
...
```
This will add a container to your workload definition that will now stream the contents of (in this example) `/path/to/your/log/file.log` to `stdout`.
This log stream is then automatically collected according to any _Flows_ or _ClusterFlows_ you have setup. You may also wish to consider creating a _Flow_ specifically for this log file by targeting the name of the container. See example:
```
...
spec:
match:
- select:
container_names:
- stream-log-file-name
...
```
# General Best Practices
- Where possible, output structured log entries (e.g. `syslog`, JSON). This makes handling of the log entry easier as there are already parsers written for these formats.
- Try to provide the name of the application that is creating the log entry, in the entry itself. This can make troubleshooting easier as Kubernetes objects do not always carry the name of the application as the object name. For instance, a pod ID may be something like `myapp-098kjhsdf098sdf98` which does not provide much information about the application running inside the container.
- Except in the case of collecting all logs cluster-wide, try to scope your _Flow_ and _ClusterFlow_ objects tightly. This makes it easier to troubleshoot when problems arise, and also helps ensure unrelated log entries do not show up in your aggregator. An example of tight scoping would be to constrain a _Flow_ to a single _Deployment_ in a namespace, or perhaps even a single container within a _Pod_.
- Keep the log verbosity down except when troubleshooting. High log verbosity poses a number of issues, chief among them being **noise**: significant events can be drowned out in a sea of `DEBUG` messages. This is somewhat mitigated with automated alerting and scripting, but highly verbose logging still places an inordinate amount of stress on the logging infrastructure.
- Where possible, try to provide a transaction or request ID with the log entry. This can make tracing application activity across multiple log sources easier, especially when dealing with distributed applications.
docs/reference-guides/best-practices/rancher-managed-clusters/monitoring-best-practices.md
+120
View File
@@ -0,0 +1,120 @@
---
title: Monitoring Best Practices
weight: 2
---
Configuring sensible monitoring and alerting rules is vital for running any production workloads securely and reliably. This is not different when using Kubernetes and Rancher. Fortunately the integrated monitoring and alerting functionality makes this whole process a lot easier.
The [Rancher monitoring documentation](../../../pages-for-subheaders/monitoring-and-alerting.md) describes how you can set up a complete Prometheus and Grafana stack. Out of the box this will scrape monitoring data from all system and Kubernetes components in your cluster and provide sensible dashboards and alerts for them to get started. But for a reliable setup, you also need to monitor your own workloads and adapt Prometheus and Grafana to your own specific use cases and cluster sizes. This document aims to give you best practices for this.
- [What to Monitor](#what-to-monitor)
- [Configuring Prometheus Resource Usage](#configuring-prometheus-resource-usage)
- [Scraping Custom Workloads](#scraping-custom-workloads)
- [Monitoring in a (Micro)Service Architecture](#monitoring-in-a-micro-service-architecture)
- [Real User Monitoring](#real-user-monitoring)
- [Security Monitoring](#security-monitoring)
- [Setting up Alerts](#setting-up-alerts)
# What to Monitor
Kubernetes itself, as well as applications running inside of it, form a distributed system where different components interact with each other. For the whole system and each individual component, you have to ensure performance, availability, reliability and scalability. A good resource with more details and information is Google's free [Site Reliability Engineering Book](https://landing.google.com/sre/sre-book/), especially the chapter about [Monitoring distributed systems](https://landing.google.com/sre/sre-book/chapters/monitoring-distributed-systems/).
# Configuring Prometheus Resource Usage
When installing the integrated monitoring stack, Rancher allows to configure several settings that are dependent on the size of your cluster and the workloads running in it. This chapter covers these in more detail.
### Storage and Data Retention
The amount of storage needed for Prometheus directly correlates to the amount of time series and labels that you store and the data retention you have configured. It is important to note that Prometheus is not meant to be used as a long-term metrics storage. Data retention time is usually only a couple of days and not weeks or months. The reason for this is that Prometheus does not perform any aggregation on its stored metrics. This is great because aggregation can dilute data, but it also means that the needed storage grows linearly over time without retention.
One way to calculate the necessary storage is to look at the average size of a storage chunk in Prometheus with this query
```
rate(prometheus_tsdb_compaction_chunk_size_bytes_sum[1h]) / rate(prometheus_tsdb_compaction_chunk_samples_sum[1h])
```
Next, find out your data ingestion rate per second:
```
rate(prometheus_tsdb_head_samples_appended_total[1h])
```
and then multiply this with the retention time, adding a few percentage points as buffer:
```
average chunk size in bytes * ingestion rate per second * retention time in seconds * 1.1 = necessary storage in bytes
```
You can find more information about how to calculate the necessary storage in this [blog post](https://www.robustperception.io/how-much-disk-space-do-prometheus-blocks-use).
You can read more about the Prometheus storage concept in the [Prometheus documentation](https://prometheus.io/docs/prometheus/latest/storage).
### CPU and Memory Requests and Limits
In larger Kubernetes clusters Prometheus can consume quite a bit of memory. The amount of memory Prometheus needs directly correlates to the amount of time series and amount of labels it stores and the scrape interval in which these are filled.
You can find more information about how to calculate the necessary memory in this [blog post](https://www.robustperception.io/how-much-ram-does-prometheus-2-x-need-for-cardinality-and-ingestion).
The amount of necessary CPUs correlate with the amount of queries you are performing.
### Federation and Long-term Storage
Prometheus is not meant to store metrics for a long amount of time, but should only be used for short term storage.
In order to store some, or all metrics for a long time, you can leverage Prometheus' [remote read/write](https://prometheus.io/docs/prometheus/latest/storage/#remote-storage-integrations) capabilities to connect it to storage systems like [Thanos](https://thanos.io/), [InfluxDB](https://www.influxdata.com/), [M3DB](https://www.m3db.io/), or others. You can find an example setup in this [blog post](https://rancher.com/blog/2020/prometheus-metric-federation).
# Scraping Custom Workloads
While the integrated Rancher Monitoring already scrapes system metrics from a cluster's nodes and system components, the custom workloads that you deploy on Kubernetes should also be scraped for data. For that you can configure Prometheus to do an HTTP request to an endpoint of your applications in a certain interval. These endpoints should then return their metrics in a Prometheus format.
In general, you want to scrape data from all the workloads running in your cluster so that you can use them for alerts or debugging issues. Often, you recognize that you need some data only when you actually need the metrics during an incident. It is good, if it is already scraped and stored. Since Prometheus is only meant to be a short-term metrics storage, scraping and keeping lots of data is usually not that expensive. If you are using a long-term storage solution with Prometheus, you can then still decide which data you are actually persisting and keeping there.
### About Prometheus Exporters
A lot of 3rd party workloads like databases, queues or web-servers either already support exposing metrics in a Prometheus format, or there are so called exporters available that translate between the tool's metrics and the format that Prometheus understands. Usually you can add these exporters as additional sidecar containers to the workload's Pods. A lot of helm charts already include options to deploy the correct exporter. Additionally you can find a curated list of exports by SysDig on [promcat.io](https://promcat.io/) and on [ExporterHub](https://exporterhub.io/).
### Prometheus support in Programming Languages and Frameworks
To get your own custom application metrics into Prometheus, you have to collect and expose these metrics directly from your application's code. Fortunately, there are already libraries and integrations available to help with this for most popular programming languages and frameworks. One example for this is the Prometheus support in the [Spring Framework](https://docs.spring.io/spring-metrics/docs/current/public/prometheus).
### ServiceMonitors and PodMonitors
Once all your workloads expose metrics in a Prometheus format, you have to configure Prometheus to scrape it. Under the hood Rancher is using the [prometheus-operator](https://github.com/prometheus-operator/prometheus-operator). This makes it easy to add additional scraping targets with ServiceMonitors and PodMonitors. A lot of helm charts already include an option to create these monitors directly. You can also find more information in the Rancher documentation.
### Prometheus Push Gateway
There are some workloads that are traditionally hard to scrape by Prometheus. Examples for these are short lived workloads like Jobs and CronJobs, or applications that do not allow sharing data between individual handled incoming requests, like PHP applications.
To still get metrics for these use cases, you can set up [prometheus-pushgateways](https://github.com/prometheus/pushgateway). The CronJob or PHP application would push metric updates to the pushgateway. The pushgateway aggregates and exposes them through an HTTP endpoint, which then can be scraped by Prometheus.
### Prometheus Blackbox Monitor
Sometimes it is useful to monitor workloads from the outside. For this, you can use the [Prometheus blackbox-exporter](https://github.com/prometheus/blackbox_exporter) which allows probing any kind of endpoint over HTTP, HTTPS, DNS, TCP and ICMP.
# Monitoring in a (Micro)Service Architecture
If you have a (micro)service architecture where multiple individual workloads within your cluster are communicating with each other, it is really important to have detailed metrics and traces about this traffic to understand how all these workloads are communicating with each other and where a problem or bottleneck may be.
Of course you can monitor all this internal traffic in all your workloads and expose these metrics to Prometheus. But this can quickly become quite work intensive. Service Meshes like Istio, which can be installed with [a click](https://rancher.com/docs/rancher/v2.6/en/istio/) in Rancher, can do this automatically and provide rich telemetry about the traffic between all services.
# Real User Monitoring
Monitoring the availability and performance of all your internal workloads is vitally important to run stable, reliable and fast applications. But these metrics only show you parts of the picture. To get a complete view it is also necessary to know how your end users are actually perceiving it. For this you can look into various [Real user monitoring solutions](https://en.wikipedia.org/wiki/Real_user_monitoring).
# Security Monitoring
In addition to monitoring workloads to detect performance, availability or scalability problems, the cluster and the workloads running into it should also be monitored for potential security problems. A good starting point is to frequently run and alert on [CIS Scans](../../../pages-for-subheaders/cis-scan-guides.md) which check if the cluster is configured according to security best practices.
For the workloads, you can have a look at Kubernetes and Container security solutions like [Falco](https://falco.org/), [Aqua Kubernetes Security](https://www.aquasec.com/solutions/kubernetes-container-security/), [SysDig](https://sysdig.com/).
# Setting up Alerts
Getting all the metrics into a monitoring systems and visualizing them in dashboards is great, but you also want to be pro-actively alerted if something goes wrong.
The integrated Rancher monitoring already configures a sensible set of alerts that make sense in any Kubernetes cluster. You should extend these to cover your specific workloads and use cases.
When setting up alerts, configure them for all the workloads that are critical to the availability of your applications. But also make sure that they are not too noisy. Ideally every alert you are receiving should be because of a problem that needs your attention and needs to be fixed. If you have alerts that are firing all the time but are not that critical, there is a danger that you start ignoring your alerts all together and then miss the real important ones. Less may be more here. Start to focus on the real important metrics first, for example alert if your application is offline. Fix all the problems that start to pop up and then start to create more detailed alerts.
If an alert starts firing, but there is nothing you can do about it at the moment, it's also fine to silence the alert for a certain amount of time, so that you can look at it later.
You can find more information on how to set up alerts and notification channels in the [Rancher Documentation](../../../pages-for-subheaders/monitoring-and-alerting.md).
docs/reference-guides/best-practices/rancher-managed-clusters/rancher-managed-clusters-in-vsphere.md
+59
View File
@@ -0,0 +1,59 @@
---
title: Best Practices for Rancher Managed vSphere Clusters
shortTitle: Rancher Managed Clusters in vSphere
---
This guide outlines a reference architecture for provisioning downstream Rancher clusters in a vSphere environment, in addition to standard vSphere best practices as documented by VMware.
- [1. VM Considerations](#1-vm-considerations)
- [2. Network Considerations](#2-network-considerations)
- [3. Storage Considerations](#3-storage-considerations)
- [4. Backups and Disaster Recovery](#4-backups-and-disaster-recovery)
<figcaption>Solution Overview</figcaption>
![Solution Overview](/img/solution_overview.drawio.svg)
# 1. VM Considerations
### Leverage VM Templates to Construct the Environment
To facilitate consistency across the deployed Virtual Machines across the environment, consider the use of "Golden Images" in the form of VM templates. Packer can be used to accomplish this, adding greater customisation options.
### Leverage DRS Anti-Affinity Rules (Where Possible) to Separate Downstream Cluster Nodes Across ESXi Hosts
Doing so will ensure node VM's are spread across multiple ESXi hosts - preventing a single point of failure at the host level.
### Leverage DRS Anti-Affinity Rules (Where Possible) to Separate Downstream Cluster Nodes Across Datastores
Doing so will ensure node VM's are spread across multiple datastores - preventing a single point of failure at the datastore level.
### Configure VM's as Appropriate for Kubernetes
It’s important to follow K8s and etcd best practices when deploying your nodes, including disabling swap, double-checking you have full network connectivity between all machines in the cluster, using unique hostnames, MAC addresses, and product_uuids for every node.
# 2. Network Considerations
### Leverage Low Latency, High Bandwidth Connectivity Between ETCD Nodes
Deploy etcd members within a single data center where possible to avoid latency overheads and reduce the likelihood of network partitioning. For most setups, 1Gb connections will suffice. For large clusters, 10Gb connections can reduce the time taken to restore from backup.
### Consistent IP Addressing for VM's
Each node used should have a static IP configured. In the case of DHCP, each node should have a DHCP reservation to make sure the node gets the same IP allocated.
# 3. Storage Considerations
### Leverage SSD Drives for ETCD Nodes
ETCD is very sensitive to write latency. Therefore, leverage SSD disks where possible.
# 4. Backups and Disaster Recovery
### Perform Regular Downstream Cluster Backups
Kubernetes uses etcd to store all its data - from configuration, state and metadata. Backing this up is crucial in the event of disaster recovery.
### Back up Downstream Node VMs
Incorporate the Rancher downstream node VM's within a standard VM backup policy.
docs/reference-guides/best-practices/rancher-managed-clusters/tips-to-set-up-containers.md
+53
View File
@@ -0,0 +1,53 @@
---
title: Tips for Setting Up Containers
weight: 100
---
Running well-built containers can greatly impact the overall performance and security of your environment.
Below are a few tips for setting up your containers.
For a more detailed discussion of security for containers, you can also refer to Rancher's [Guide to Container Security.](https://rancher.com/complete-guide-container-security)
### Use a Common Container OS
When possible, you should try to standardize on a common container base OS.
Smaller distributions such as Alpine and BusyBox reduce container image size and generally have a smaller attack/vulnerability surface.
Popular distributions such as Ubuntu, Fedora, and CentOS are more field-tested and offer more functionality.
### Start with a FROM scratch container
If your microservice is a standalone static binary, you should use a FROM scratch container.
The FROM scratch container is an [official Docker image](https://hub.docker.com/_/scratch) that is empty so that you can use it to design minimal images.
This will have the smallest attack surface and smallest image size.
### Run Container Processes as Unprivileged
When possible, use a non-privileged user when running processes within your container. While container runtimes provide isolation, vulnerabilities and attacks are still possible. Inadvertent or accidental host mounts can also be impacted if the container is running as root. For details on configuring a security context for a pod or container, refer to the [Kubernetes docs](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/).
### Define Resource Limits
Apply CPU and memory limits to your pods. This can help manage the resources on your worker nodes and avoid a malfunctioning microservice from impacting other microservices.
In standard Kubernetes, you can set resource limits on the namespace level. In Rancher, you can set resource limits on the project level and they will propagate to all the namespaces within the project. For details, refer to the Rancher docs.
When setting resource quotas, if you set anything related to CPU or Memory (i.e. limits or reservations) on a project or namespace, all containers will require a respective CPU or Memory field set during creation. To avoid setting these limits on each and every container during workload creation, a default container resource limit can be specified on the namespace.
The Kubernetes docs have more information on how resource limits can be set at the [container level](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#resource-requests-and-limits-of-pod-and-container) and the namespace level.
### Define Resource Requirements
You should apply CPU and memory requirements to your pods. This is crucial for informing the scheduler which type of compute node your pod needs to be placed on, and ensuring it does not over-provision that node. In Kubernetes, you can set a resource requirement by defining `resources.requests` in the resource requests field in a pod's container spec. For details, refer to the [Kubernetes docs](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#resource-requests-and-limits-of-pod-and-container).
:::note
If you set a resource limit for the namespace that the pod is deployed in, and the container doesn't have a specific resource request, the pod will not be allowed to start. To avoid setting these fields on each and every container during workload creation, a default container resource limit can be specified on the namespace.
:::
It is recommended to define resource requirements on the container level because otherwise, the scheduler makes assumptions that will likely not be helpful to your application when the cluster experiences load.
### Liveness and Readiness Probes
Set up liveness and readiness probes for your container. Unless your container completely crashes, Kubernetes will not know it's unhealthy unless you create an endpoint or mechanism that can report container status. Alternatively, make sure your container halts and crashes if unhealthy.
The Kubernetes docs show how to [configure liveness and readiness probes for containers.](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/)
docs/reference-guides/best-practices/rancher-server/on-premises-rancher-in-vsphere.md
+91
View File
@@ -0,0 +1,91 @@
---
title: Installing Rancher in a vSphere Environment
shortTitle: On-Premises Rancher in vSphere
weight: 3
---
This guide outlines a reference architecture for installing Rancher on an RKE Kubernetes cluster in a vSphere environment, in addition to standard vSphere best practices as documented by VMware.
- [1. Load Balancer Considerations](#1-load-balancer-considerations)
- [2. VM Considerations](#2-vm-considerations)
- [3. Network Considerations](#3-network-considerations)
- [4. Storage Considerations](#4-storage-considerations)
- [5. Backups and Disaster Recovery](#5-backups-and-disaster-recovery)
<figcaption>Solution Overview</figcaption>
![Solution Overview](/img/rancher-on-prem-vsphere.svg)
# 1. Load Balancer Considerations
A load balancer is required to direct traffic to the Rancher workloads residing on the RKE nodes.
### Leverage Fault Tolerance and High Availability
Leverage the use of an external (hardware or software) load balancer that has inherit high-availability functionality (F5, NSX-T, Keepalived, etc).
### Back Up Load Balancer Configuration
In the event of a Disaster Recovery activity, availability of the Load balancer configuration will expedite the recovery process.
### Configure Health Checks
Configure the Load balancer to automatically mark nodes as unavailable if a health check is failed. For example, NGINX can facilitate this with:
`max_fails=3 fail_timeout=5s`
### Leverage an External Load Balancer
Avoid implementing a software load balancer within the management cluster.
### Secure Access to Rancher
Configure appropriate Firewall / ACL rules to only expose access to Rancher
# 2. VM Considerations
### Size the VM's According to Rancher Documentation
https://rancher.com/docs/rancher/v2.6/en/installation/requirements/
### Leverage VM Templates to Construct the Environment
To facilitate the consistency of Virtual Machines deployed across the environment, consider the use of "Golden Images" in the form of VM templates. Packer can be used to accomplish this, adding greater customization options.
### Leverage DRS Anti-Affinity Rules (Where Possible) to Separate Rancher Cluster Nodes Across ESXi Hosts
Doing so will ensure node VM's are spread across multiple ESXi hosts - preventing a single point of failure at the host level.
### Leverage DRS Anti-Affinity Rules (Where Possible) to Separate Rancher Cluster Nodes Across Datastores
Doing so will ensure node VM's are spread across multiple datastores - preventing a single point of failure at the datastore level.
### Configure VM's as Appropriate for Kubernetes
It’s important to follow K8s and etcd best practices when deploying your nodes, including disabling swap, double-checking you have full network connectivity between all machines in the cluster, using unique hostnames, MAC addresses, and product_uuids for every node.
# 3. Network Considerations
### Leverage Low Latency, High Bandwidth Connectivity Between ETCD Nodes
Deploy etcd members within a single data center where possible to avoid latency overheads and reduce the likelihood of network partitioning. For most setups, 1Gb connections will suffice. For large clusters, 10Gb connections can reduce the time taken to restore from backup.
### Consistent IP Addressing for VM's
Each node used should have a static IP configured. In the case of DHCP, each node should have a DHCP reservation to make sure the node gets the same IP allocated.
# 4. Storage Considerations
### Leverage SSD Drives for ETCD Nodes
ETCD is very sensitive to write latency. Therefore, leverage SSD disks where possible.
# 5. Backups and Disaster Recovery
### Perform Regular Management Cluster Backups
Rancher stores its data in the ETCD datastore of the Kubernetes cluster it resides on. Like with any Kubernetes cluster, perform frequent, tested backups of this cluster.
### Back up Rancher Cluster Node VMs
Incorporate the Rancher management node VM's within a standard VM backup policy.
docs/reference-guides/best-practices/rancher-server/rancher-deployment-strategy.md
+45
View File
@@ -0,0 +1,45 @@
---
title: Rancher Deployment Strategy
weight: 100
---
There are two recommended deployment strategies for a Rancher server that manages downstream Kubernetes clusters. Each one has its own pros and cons. Read more about which one would fit best for your use case:
* [Hub and Spoke](#hub-and-spoke-strategy)
* [Regional](#regional-strategy)
# Hub & Spoke Strategy
---
In this deployment scenario, there is a single Rancher control plane managing Kubernetes clusters across the globe. The control plane would be run on a high-availability Kubernetes cluster, and there would be impact due to latencies.
![](/img/bpg/hub-and-spoke.png)
### Pros
* Environments could have nodes and network connectivity across regions.
* Single control plane interface to view/see all regions and environments.
* Kubernetes does not require Rancher to operate and can tolerate losing connectivity to the Rancher control plane.
### Cons
* Subject to network latencies.
* If the control plane goes out, global provisioning of new services is unavailable until it is restored. However, each Kubernetes cluster can continue to be managed individually.
# Regional Strategy
---
In the regional deployment model a control plane is deployed in close proximity to the compute nodes.
![](/img/bpg/regional.png)
### Pros
* Rancher functionality in regions stay operational if a control plane in another region goes down.
* Network latency is greatly reduced, improving the performance of functionality in Rancher.
* Upgrades of the Rancher control plane can be done independently per region.
### Cons
* Overhead of managing multiple Rancher installations.
* Visibility across global Kubernetes clusters requires multiple interfaces/panes of glass.
* Deploying multi-cluster apps in Rancher requires repeating the process for each Rancher server.
docs/reference-guides/best-practices/rancher-server/tips-for-running-rancher.md
+37
View File
@@ -0,0 +1,37 @@
---
title: Tips for Running Rancher
weight: 100
---
This guide is geared toward use cases where Rancher is used to manage downstream Kubernetes clusters. The high-availability setup is intended to prevent losing access to downstream clusters if the Rancher server is not available.
A high-availability Kubernetes installation, defined as an installation of Rancher on a Kubernetes cluster with at least three nodes, should be used in any production installation of Rancher, as well as any installation deemed "important." Multiple Rancher instances running on multiple nodes ensure high availability that cannot be accomplished with a single node environment.
If you are installing Rancher in a vSphere environment, refer to the best practices documented [here.](on-premises-rancher-in-vsphere.md)
When you set up your high-availability Rancher installation, consider the following:
### Run Rancher on a Separate Cluster
Don't run other workloads or microservices in the Kubernetes cluster that Rancher is installed on.
### Make sure nodes are configured correctly for Kubernetes ###
It's important to follow K8s and etcd best practices when deploying your nodes, including disabling swap, double checking you have full network connectivity between all machines in the cluster, using unique hostnames, MAC addresses, and product_uuids for every node, checking that all correct ports are opened, and deploying with ssd backed etcd. More details can be found in the [kubernetes docs](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#before-you-begin) and [etcd's performance op guide](https://etcd.io/docs/v3.4/op-guide/performance/).
### When using RKE: Back up the Statefile
RKE keeps record of the cluster state in a file called `cluster.rkestate`. This file is important for the recovery of a cluster and/or the continued maintenance of the cluster through RKE. Because this file contains certificate material, we strongly recommend encrypting this file before backing up. After each run of `rke up` you should backup the state file.
### Run All Nodes in the Cluster in the Same Datacenter
For best performance, run all three of your nodes in the same geographic datacenter. If you are running nodes in the cloud, such as AWS, run each node in a separate Availability Zone. For example, launch node 1 in us-west-2a, node 2 in us-west-2b, and node 3 in us-west-2c.
### Development and Production Environments Should be Similar
It's strongly recommended to have a "staging" or "pre-production" environment of the Kubernetes cluster that Rancher runs on. This environment should mirror your production environment as closely as possible in terms of software and hardware configuration.
### Monitor Your Clusters to Plan Capacity
The Rancher server's Kubernetes cluster should run within the [system and hardware requirements](../../../pages-for-subheaders/installation-requirements.md) as closely as possible. The more you deviate from the system and hardware requirements, the more risk you take.
However, metrics-driven capacity planning analysis should be the ultimate guidance for scaling Rancher, because the published requirements take into account a variety of workload types.
Using Rancher, you can monitor the state and processes of your cluster nodes, Kubernetes components, and software deployments through integration with Prometheus, a leading open-source monitoring solution, and Grafana, which lets you visualize the metrics from Prometheus.
After you [enable monitoring](../../../pages-for-subheaders/monitoring-and-alerting.md) in the cluster, you can set up alerts to let you know if your cluster is approaching its capacity. You can also use the Prometheus and Grafana monitoring framework to establish a baseline for key metrics as you scale.
docs/reference-guides/cli-with-rancher/kubectl-utility.md
+1
View File
@@ -0,0 +1 @@
<!-- PLACEHOLDER -->
docs/reference-guides/cli-with-rancher/rancher-cli.md
+1
View File
@@ -0,0 +1 @@
<!-- PLACEHOLDER -->
docs/reference-guides/cluster-configuration/downstream-cluster-configuration/machine-configuration/amazon-ec2.md
+79
View File
@@ -0,0 +1,79 @@
---
title: EC2 Machine Configuration Reference
weight: 2
---
For more details about EC2 nodes, refer to the official documentation for the [EC2 Management Console](https://aws.amazon.com/ec2).
### Region
The geographical [region](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html) in which to build your cluster.
### Zone
The [zone](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html#concepts-availability-zones), an isolated location within a region to build your cluster
### Instance Type
The [instance type](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-types.html), which determines the hardware characteristics, used to provision your cluster.
### Root Disk Size
Configure the size (in GB) for your [root device](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/RootDeviceStorage.html).
### VPC/Subnet
The [VPC](https://docs.aws.amazon.com/vpc/latest/userguide/configure-your-vpc.html) or specific [subnet](https://docs.aws.amazon.com/vpc/latest/userguide/configure-subnets.html), an IP range in your VPC, to add your resources to.
### IAM Instance Profile Name
The name of the [instance profile] used to pass an IAM role to an EC2 instance.
## Advanced Options
### AMI ID
The [Amazon Machine Image](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIs.html) used for the nodes in your cluster.
### SSH Username for AMI
The username for connecting to your launched instances. Refer to [here](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/connection-prereqs.html) for the default usernames to selected AMIs. For AMIs not listed, check with the AMI provider.
### Security Group
Choose the default security group or configure a security group.
Please refer to [Amazon EC2 security group when using Node Driver](../../../../getting-started/installation-and-upgrade/installation-requirements/port-requirements.md#rancher-aws-ec2-security-group) to see what rules are created in the `rancher-nodes` Security Group.
### EBS Root Volume Type
The [EBS volume type](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-volume-types.html) to use for the root device.
### Encrypt EBS Volume
Enable [Amazon EBS Encryption](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSEncryption.html).
### Request Spot Instance
Enable option to [request spot instances](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-requests.html) and specify the maximum instance price per hour you're willing to pay.
### Use only private address
Enable option on use only [private addresses](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-instance-addressing.html).
### EBS-Optimized Instance
Use an [EBS-optimized instance](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-optimized.html).
### Allow access to EC2 metadata
Enable access to [EC2 metadata](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html).
### Use tokens for metadata
Use [Instance Metadata Service Version 2 (IMDSv2)](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/configuring-instance-metadata-service.html), a token-based method to access metadata.
### Add Tag
Add metadata using [tags](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_Tags.html) to categorize resources.
docs/reference-guides/cluster-configuration/downstream-cluster-configuration/machine-configuration/azure.md
+121
View File
@@ -0,0 +1,121 @@
---
title: Azure Machine Configuration
weight: 2
---
For more information about Azure, refer to the official [Azure documentation.](https://docs.microsoft.com/en-us/azure/?product=featured)
### Environment
Microsoft provides multiple [clouds](https://docs.microsoft.com/en-us/cli/azure/cloud?view=azure-cli-latest) for compliance with regional laws, which are available for your use:
- AzurePublicCloud
- AzureGermanCloud
- AzureChinaCloud
- AzureUSGovernmentCloud
### Location
Configure the cluster and node [location](https://docs.microsoft.com/en-us/azure/virtual-machines/regions).
### Resource Group
A resource group is a container that holds related resources for an Azure solution. The resource group can include all the resources for the solution, or only those resources that you want to manage as a group. You decide how you want to allocate resources to resource groups based on what makes the most sense for your organization. Generally, add resources that share the same lifecycle to the same resource group so you can easily deploy, update, and delete them as a group.
Use an existing resource group or enter a resource group name and one will be created for you.
For information on managing resource groups, see the [Azure documentation.](https://docs.microsoft.com/en-us/azure/azure-resource-manager/management/manage-resource-groups-portal)
### Availability Set (unmanaged)
Name or ID of an existing [availability set](https://docs.microsoft.com/en-us/azure/virtual-machines/availability-set-overview) to add the VM to.
### Image
The name of the operating system image provided as an ARM resource identifier. Requires using managed disk.
### VM Size
Choose a size for each VM in the node pool. For details about each VM size, see [this page.](https://azure.microsoft.com/en-us/pricing/details/virtual-machines/linux/)
## Advanced Options
### Fault Domain Count
Fault domains define the group of virtual machines that share a common power source and network switch. If the availability set has already been created, the fault domain count will be ignored.
For more information on fault domains, see [refer here](https://docs.microsoft.com/en-us/azure/virtual-machines/availability-set-overview#how-do-availability-sets-work).
### Update Domain Count
Update domains indicate groups of virtual machines and underlying physical hardware that can be rebooted at the same time. If the availability set has already been created, the update domain count will be ignored.
For more information on update domains, see [refer here](https://docs.microsoft.com/en-us/azure/virtual-machines/availability-set-overview#how-do-availability-sets-work).
### Purchase Plan
Some VM images in the Azure Marketplace require a plan. If applicable, select a purchase plan, formatted as `publisher:product:plan`, to use with your chosen image.
### Subnet
The name of the subnet when creating a new VNet or referencing an existing one.
Default: `docker-machine`
### Subnet Prefix
The subnet IP address prefix to use when creating a new VNet in CIDR format.
Default: `192.168.0.0/16`
### Virtual Network
The [virtual network](https://docs.microsoft.com/en-us/azure/virtual-network/virtual-networks-overview) to use or create if one does not exist. Formatted as `[resourcegroup:]name`.
### Public IP Options
#### No Public IP
Do not allocate a public IP address.
#### Static Public IP
Allocate a static public IP address.
### Use Private IP
Use a static private IP address.
### Private IP Address
Configure a static private IP address to use.
### Network Security Group
The [network security group](https://docs.microsoft.com/en-us/azure/virtual-network/network-security-groups-overview) to use. All nodes using this template will use the supplied network security group. If no network security group is provided, a new one will be created for each node.
### DNS Label
A unique DNS name label for the public IP address.
### Storage Type
The [storage account](https://docs.microsoft.com/en-us/azure/storage/common/storage-account-overview) type to use with your VMs. Options include Standard LRS, Standard ZRS, Standard GRS, Standard RAGRS, and Premium LRS.
### Use Managed Disks
[Azure managed disks](https://docs.microsoft.com/en-us/azure/virtual-machines/managed-disks-overview) are block-level storage volumes that are managed by Azure and used with Azure Virtual Machines. Managed disks are designed for 99.999% availability. Managed disks achieve this by providing you with three replicas of your data, allowing for high durability.
### Managed Disk Size
The size in GB for the disk for each node.
### SSH Username
The username used to create an SSH connection to your nodes.
### Open Port
Opens inbound traffic on specified ports. When using an existing Network Security Group, Open Ports are ignored.
Default: `2379/tcp, 2380/tcp, 6443/tcp, 9796/tcp, 10250/tcp, 10251/tcp, 10252/tcp, 10256/tcp` and `8472/udp, 4789/udp`
docs/reference-guides/cluster-configuration/downstream-cluster-configuration/machine-configuration/digitalocean.md
+34
View File
@@ -0,0 +1,34 @@
---
title: DigitalOcean Machine Configuration
weight: 2
---
For more details about DigitalOcean, Droplets, refer to the [official documentation](https://docs.digitalocean.com/products/compute/).
### Region
Configure the [region](https://docs.digitalocean.com/products/app-platform/concepts/region/) where Droplets are created.
### Size
Configure the [size](https://docs.digitalocean.com/products/droplets/resources/choose-plan/) of Droplets.
### OS Image
Configure the operating system [image](https://docs.digitalocean.com/products/images/) Droplets are created from.
### Monitoring
Enable the DigitalOcean agent for additional [monitoring](https://docs.digitalocean.com/products/monitoring/).
### IPv6
Enable IPv6 for Droplets.
### Private Networking
Enable private networking for Droplets.
### Droplet Tags
Apply a tag (label) to a Droplet. Tags may only contain letters, numbers, colons, dashes, and underscores. For example, `my_server`.
docs/reference-guides/cluster-configuration/downstream-cluster-configuration/node-template-configuration/amazon-ec2.md
+50
View File
@@ -0,0 +1,50 @@
---
title: EC2 Node Template Configuration
weight: 1
---
For more details about EC2, nodes, refer to the official documentation for the [EC2 Management Console](https://aws.amazon.com/ec2).
### Region
In the **Region** field, select the same region that you used when creating your cloud credentials.
### Cloud Credentials
Your AWS account access information, stored in a [cloud credential.](../../../user-settings/manage-cloud-credentials.md)
See [Amazon Documentation: Creating Access Keys](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html#Using_CreateAccessKey) how to create an Access Key and Secret Key.
See [Amazon Documentation: Creating IAM Policies (Console)](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_create.html#access_policies_create-start) how to create an IAM policy.
See [Amazon Documentation: Adding Permissions to a User (Console)](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_change-permissions.html#users_change_permissions-add-console) how to attach an IAM
See our three example JSON policies:
- [Example IAM Policy](../../../../how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/create-an-amazon-ec2-cluster.md#example-iam-policy)
- [Example IAM Policy with PassRole](../../../../how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/create-an-amazon-ec2-cluster.md#example-iam-policy-with-passrole) (needed if you want to use [Kubernetes Cloud Provider](../../../../pages-for-subheaders/set-up-cloud-providers.md) or want to pass an IAM Profile to an instance)
- [Example IAM Policy to allow encrypted EBS volumes](../../../../how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/create-an-amazon-ec2-cluster.md#example-iam-policy-to-allow-encrypted-ebs-volumes) policy to an user.
### Authenticate & Configure Nodes
Choose an availability zone and network settings for your cluster.
### Security Group
Choose the default security group or configure a security group.
Please refer to [Amazon EC2 security group when using Node Driver](../../../../getting-started/installation-and-upgrade/installation-requirements/port-requirements.md#rancher-aws-ec2-security-group) to see what rules are created in the `rancher-nodes` Security Group.
---
**_New in v2.6.4_**
If you provide your own security group for an EC2 instance, please note that Rancher will not modify it. As such, you will be responsible for ensuring that your security group is set to allow the [necessary ports for Rancher to provision the instance](../../../../getting-started/installation-and-upgrade/installation-requirements/port-requirements.md#ports-for-rancher-server-nodes-on-rke). For more information on controlling inbound and outbound traffic to EC2 instances with security groups, refer [here](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_SecurityGroups.html#WorkingWithSecurityGroups).
### Instance Options
Configure the instances that will be created. Make sure you configure the correct **SSH User** for the configured AMI. It is possible that a selected region does not support the default instance type. In this scenario you must select an instance type that does exist, otherwise an error will occur stating the requested configuration is not supported.
If you need to pass an **IAM Instance Profile Name** (not ARN), for example, when you want to use a [Kubernetes Cloud Provider](../../../../pages-for-subheaders/set-up-cloud-providers.md), you will need an additional permission in your policy. See [Example IAM policy with PassRole](#example-iam-policy-with-passrole) for an example policy.
### Engine Options
In the **Engine Options** section of the node template, you can configure the Docker daemon. You may want to specify the docker version or a Docker registry mirror.
docs/reference-guides/cluster-configuration/downstream-cluster-configuration/node-template-configuration/azure.md
+19
View File
@@ -0,0 +1,19 @@
---
title: Azure Node Template Configuration
weight: 1
---
For more information about Azure, refer to the official [Azure documentation.](https://docs.microsoft.com/en-us/azure/?product=featured)
Account access information is stored as a cloud credential. Cloud credentials are stored as Kubernetes secrets. Multiple node templates can use the same cloud credential. You can use an existing cloud credential or create a new one.
- **Placement** sets the geographical region where your cluster is hosted and other location metadata.
- **Network** configures the networking used in your cluster.
- **Instance** customizes your VM configuration.
The [Docker daemon](https://docs.docker.com/engine/docker-overview/#the-docker-daemon) configuration options include:
- **Labels:** For information on labels, refer to the [Docker object label documentation.](https://docs.docker.com/config/labels-custom-metadata/)
- **Docker Engine Install URL:** Determines what Docker version will be installed on the instance.
- **Registry mirrors:** Docker Registry mirror to be used by the Docker daemon
- **Other advanced options:** Refer to the [Docker daemon option reference](https://docs.docker.com/engine/reference/commandline/dockerd/)
docs/reference-guides/cluster-configuration/downstream-cluster-configuration/node-template-configuration/digitalocean.md
+19
View File
@@ -0,0 +1,19 @@
---
title: DigitalOcean Node Template Configuration
weight: 1
----
Account access information is stored as a cloud credential. Cloud credentials are stored as Kubernetes secrets. Multiple node templates can use the same cloud credential. You can use an existing cloud credential or create a new one.
### Droplet Options
The **Droplet Options** provision your cluster's geographical region and specifications.
### Docker Daemon
The [Docker daemon](https://docs.docker.com/engine/docker-overview/#the-docker-daemon) configuration options include:
- **Labels:** For information on labels, refer to the [Docker object label documentation.](https://docs.docker.com/config/labels-custom-metadata/)
- **Docker Engine Install URL:** Determines what Docker version will be installed on the instance.
- **Registry mirrors:** Docker Registry mirror to be used by the Docker daemon
- **Other advanced options:** Refer to the [Docker daemon option reference](https://docs.docker.com/engine/reference/commandline/dockerd/)
docs/reference-guides/cluster-configuration/downstream-cluster-configuration/node-template-configuration/nutanix.md
+64
View File
@@ -0,0 +1,64 @@
---
title: Nutanix Node Template Configuration
weight: 2
---
- [Account Access](#account-access)
- [Scheduling](#scheduling)
- [Instance Options](#instance-options)
- [Networks](#networks)
- [VM Categories](#vm-categories)
- [cloud-init](#cloud-init)
# Account Access
| Parameter | Required | Description | Default
|:-----------------------------|:--------:|:-----------------------------------------------------------------|:-----
| Management Endpoint | ✓ | Hostname/IP address of Prism Central |
| Username | ✓ | Username of the Prism Central user |
| Password | ✓ | Password of the Prism Central user |
| Allow insecure communication | | Set to true to allow insecure SSL communication to Prism Central | False
# Scheduling
Choose what Nutanix cluster the virtual machine will be scheduled to.
| Parameter | Required | Description
|:----------|:--------:|:----------------------------------------------------------------------------
| Cluster | ✓ | Name of the Nutanix cluster where the VM should be deployed (case sensitive)
# Instance Options
In the **Instance Options** section, configure the number of vCPUs, memory, and disk size for the VMs created by this template.
| Parameter | Required | Description | Default
|:---------------------|:--------:|:--------------------------------------------------------------------------------------------|:-------
| CPUs | | Number of vCPUs allocated to the VM (cores) | 2
| Memory | | Amount of RAM allocated to the VM (MB) | 2 GB
| Template Image | ✓ | Name of the Disk Image template to clone as the VM's primary disk (must support cloud-init) |
| VM Disk Size | | New size of the VM's primary disk (in GiB) |
| Additional Disk Size | | Size of an additional disk to add to the VM (in GiB) |
| Storage Container | | Storage container _UUID_ in which to provision an additional disk |
| Cloud Config YAML | | Cloud-init to provide to the VM (will be patched with Rancher root user) |
| Network | ✓ | Name(s) of the network(s) to attach to the VM |
| VM Categories | | Name(s) of any categories to be applied to the VM |
The VM may use any modern Linux operating system that is configured with support for [cloud-init](https://cloudinit.readthedocs.io/en/latest/) using the [Config Drive v2 datasource](https://cloudinit.readthedocs.io/en/latest/topics/datasources/configdrive.html).
# Networks
The node template allows a VM to be provisioned with multiple networks. In the **Network** field, you can click **Add** to add any networks available to you in AOS.
# VM Categories
A category is a grouping of entities into a key value pair. Typically, VMs are assigned to a category based on some criteria. Policies can then be tied to those entities that are assigned (grouped by) a specific category value.
# cloud-init
[Cloud-init](https://cloudinit.readthedocs.io/en/latest/) allows you to initialize your nodes by applying configuration on the first boot. This may involve things such as creating users or authorizing SSH keys.
To make use of cloud-init initialization, paste a cloud config using valid YAML syntax into the **Cloud Config YAML** field. Refer to the [cloud-init documentation](https://cloudinit.readthedocs.io/en/latest/topics/examples.html) for a commented set of examples of supported cloud config directives.
Note that cloud-init based network configuration is not recommended and only supported via user data `runcmd` rather than by NoCloud or other network configuration datasources.
Nutanix IP Address Management (IPAM) or another DHCP service is recommended.
docs/reference-guides/cluster-configuration/downstream-cluster-configuration/node-template-configuration/vsphere.md
+93
View File
@@ -0,0 +1,93 @@
---
title: VSphere Node Template Configuration
weight: 2
---
- [Account Access](#account-access)
- [Scheduling](#scheduling)
- [Instance Options](#instance-options)
- [Networks](#networks)
- [Node tags and custom attributes](#node-tags-and-custom-attributes)
- [cloud-init](#cloud-init)
# Account Access
| Parameter | Required | Description |
|:----------------------|:--------:|:-----|
| Cloud Credentials | * | Your vSphere account access information, stored in a [cloud credential.](../../../user-settings/manage-cloud-credentials.md) |
Your cloud credential has these fields:
| Credential Field | Description |
|-----------------|--------------|
| vCenter or ESXi Server | Enter the vCenter or ESXi hostname/IP. ESXi is the virtualization platform where you create and run virtual machines and virtual appliances. vCenter Server is the service through which you manage multiple hosts connected in a network and pool host resources. |
| Port | Optional: configure configure the port of the vCenter or ESXi server. |
| Username and password | Enter your vSphere login username and password. |
# Scheduling
Choose what hypervisor the virtual machine will be scheduled to.
The fields in the **Scheduling** section should auto-populate with the data center and other scheduling options that are available to you in vSphere.
| Field | Required | Explanation |
|---------|---------------|-----------|
| Data Center | * | Choose the name/path of the data center where the VM will be scheduled. |
| Resource Pool | | Name of the resource pool to schedule the VMs in. Resource pools can be used to partition available CPU and memory resources of a standalone host or cluster, and they can also be nested. Leave blank for standalone ESXi. If not specified, the default resource pool is used. |
| Data Store | * | If you have a data store cluster, you can toggle the **Data Store** field. This lets you select a data store cluster where your VM will be scheduled to. If the field is not toggled, you can select an individual disk. |
| Folder | | Name of a folder in the datacenter to create the VMs in. Must already exist. The VM folders in this dropdown menu directly correspond to your VM folders in vSphere. The folder name should be prefaced with `vm/` in your vSphere config file. |
| Host | | The IP of the host system to schedule VMs in. Leave this field blank for a standalone ESXi or for a cluster with DRS (Distributed Resource Scheduler). If specified, the host system's pool will be used and the **Resource Pool** parameter will be ignored. |
# Instance Options
In the **Instance Options** section, configure the number of vCPUs, memory, and disk size for the VMs created by this template.
| Parameter | Required | Description |
|:----------------|:--------:|:-----------|
| CPUs | * | Number of vCPUS to assign to VMs. |
| Memory | * | Amount of memory to assign to VMs. |
| Disk | * | Size of the disk (in MB) to attach to the VMs. |
| Creation method | * | The method for setting up an operating system on the node. The operating system can be installed from an ISO or from a VM template. Depending on the creation method, you will also have to specify a VM template, content library, existing VM, or ISO. For more information on creation methods, refer to [About VM Creation Methods.](#about-vm-creation-methods) |
| Cloud Init | | URL of a `cloud-config.yml` file or URL to provision VMs with. This file allows further customization of the operating system, such as network configuration, DNS servers, or system daemons. The operating system must support `cloud-init`. |
| Networks | | Name(s) of the network to attach the VM to. |
| Configuration Parameters used for guestinfo | | Additional configuration parameters for the VMs. These correspond to the [Advanced Settings](https://kb.vmware.com/s/article/1016098) in the vSphere console. Example use cases include providing RancherOS [guestinfo]({{<baseurl>}}/os/v1.x/en/installation/cloud/vmware-esxi/#vmware-guestinfo) parameters or enabling disk UUIDs for the VMs (`disk.EnableUUID=TRUE`). |
### About VM Creation Methods
In the **Creation method** field, configure the method used to provision VMs in vSphere. Available options include creating VMs that boot from a RancherOS ISO or creating VMs by cloning from an existing virtual machine or [VM template](https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.vm_admin.doc/GUID-F7BF0E6B-7C4F-4E46-8BBF-76229AEA7220.html).
The existing VM or template may use any modern Linux operating system that is configured with support for [cloud-init](https://cloudinit.readthedocs.io/en/latest/) using the [NoCloud datasource](https://cloudinit.readthedocs.io/en/latest/topics/datasources/nocloud.html).
Choose the way that the VM will be created:
- **Deploy from template: Data Center:** Choose a VM template that exists in the data center that you selected.
- **Deploy from template: Content Library:** First, select the [Content Library](https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.vm_admin.doc/GUID-254B2CE8-20A8-43F0-90E8-3F6776C2C896.html) that contains your template, then select the template from the populated list **Library templates**.
- **Clone an existing virtual machine:** In the **Virtual machine** field, choose an existing VM that the new VM will be cloned from.
- **Install from boot2docker ISO:** Ensure that the **OS ISO URL** field contains the URL of a VMware ISO release for RancherOS (`rancheros-vmware.iso`). Note that this URL must be accessible from the nodes running your Rancher server installation.
# Networks
The node template now allows a VM to be provisioned with multiple networks. In the **Networks** field, you can now click **Add Network** to add any networks available to you in vSphere.
# Node Tags and Custom Attributes
Tags allow you to attach metadata to objects in the vSphere inventory to make it easier to sort and search for these objects.
For tags, all your vSphere tags will show up as options to select from in your node template.
In the custom attributes, Rancher will let you select all the custom attributes you have already set up in vSphere. The custom attributes are keys and you can enter values for each one.
:::note
Custom attributes are a legacy feature that will eventually be removed from vSphere.
:::
# cloud-init
[Cloud-init](https://cloudinit.readthedocs.io/en/latest/) allows you to initialize your nodes by applying configuration on the first boot. This may involve things such as creating users, authorizing SSH keys or setting up the network.
To make use of cloud-init initialization, create a cloud config file using valid YAML syntax and paste the file content in the the **Cloud Init** field. Refer to the [cloud-init documentation.](https://cloudinit.readthedocs.io/en/latest/topics/examples.html) for a commented set of examples of supported cloud config directives.
Note that cloud-init is not supported when using the ISO creation method.
docs/reference-guides/cluster-configuration/rancher-server-configuration/aks-cluster-configuration.md
+226
View File
@@ -0,0 +1,226 @@
---
shortTitle: AKS Cluster Configuration
title: AKS Cluster Configuration Reference
weight: 4
---
# Changes in Rancher v2.6
- Support for adding more than one node pool
- Support for private clusters
- Enabled autoscaling node pools
- The AKS permissions are now configured in cloud credentials
# Role-based Access Control
When provisioning an AKS cluster in the Rancher UI, RBAC cannot be disabled. If role-based access control is disabled for the cluster in AKS, the cluster cannot be registered or imported into Rancher.
Rancher can configure member roles for AKS clusters in the same way as any other cluster. For more information, see the section on [role-based access control.](../../../pages-for-subheaders/manage-role-based-access-control-rbac.md)
# Cloud Credentials
:::note
The configuration information in this section assumes you have already set up a service principal for Rancher. For step-by-step instructions for how to set up the service principal, see [this section.](../../../how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/set-up-clusters-from-hosted-kubernetes-providers/aks.md#prerequisites-in-microsoft-azure)
:::
### Subscription ID
To get the subscription ID, click **All Services** in the left navigation bar. Then click **Subscriptions**. Go to the name of the subscription that you want to associate with your Kubernetes cluster and copy the **Subscription ID**.
### Client ID
To get the client ID, go to the Azure Portal, then click **Azure Active Directory**, then click **App registrations,** then click the name of the service principal. The client ID is listed on the app registration detail page as **Application (client) ID**.
### Client Secret
You can't retrieve the client secret value after it is created, so if you don't already have a client secret value, you will need to create a new client secret.
To get a new client secret, go to the Azure Portal, then click **Azure Active Directory**, then click **App registrations,** then click the name of the service principal.
Then click **Certificates & secrets** and click **New client secret**. Click **Add**. Then copy the **Value** of the new client secret.
### Environment
Microsoft provides multiple [clouds](https://docs.microsoft.com/en-us/cli/azure/cloud?view=azure-cli-latest) for compliance with regional laws, which are available for your use:
- AzurePublicCloud
- AzureGermanCloud
- AzureChinaCloud
- AzureUSGovernmentCloud
# Account Access
In this section you will need to select an existing Azure cloud credential or create a new one.
For help configuring your Azure cloud credential, see [this section.](#cloud-credentials)
# Cluster Location
Configure the cluster and node location. For more information on availability zones for AKS, see the [AKS documentation.](https://docs.microsoft.com/en-us/azure/aks/availability-zones)
The high availability locations include multiple availability zones.
# Cluster Options
### Kubernetes Version
The available Kubernetes versions are dynamically fetched from the Azure API.
### Cluster Resource Group
A resource group is a container that holds related resources for an Azure solution. The resource group can include all the resources for the solution, or only those resources that you want to manage as a group. You decide how you want to allocate resources to resource groups based on what makes the most sense for your organization. Generally, add resources that share the same lifecycle to the same resource group so you can easily deploy, update, and delete them as a group.
Use an existing resource group or enter a resource group name and one will be created for you.
Using a resource group containing an existing AKS cluster will create a new resource group. Azure AKS only allows one AKS cluster per resource group.
For information on managing resource groups, see the [Azure documentation.](https://docs.microsoft.com/en-us/azure/azure-resource-manager/management/manage-resource-groups-portal)
### Linux Admin Username
The username used to create an SSH connection to the Linux nodes.
The default username for AKS nodes is `azureuser`.
### SSH Public Key
The key used to create an SSH connection to the Linux nodes.
### Tags
Cluster tags can be useful if your organization uses tags as a way to organize resources across multiple Azure services. These tags don't apply to resources within the cluster.
# Networking Options
### LoadBalancer SKU
Azure load balancers support both standard and basic SKUs (stock keeping units).
For a comparison of standard and basic load balancers, see the official [Azure documentation.](https://docs.microsoft.com/en-us/azure/load-balancer/skus#skus) Microsoft recommends the Standard load balancer.
The Standard load balancer is required if you have selected one or more availability zones, or if you have more than one node pool.
### Network Policy
All pods in an AKS cluster can send and receive traffic without limitations, by default. To improve security, you can define rules that control the flow of traffic. The Network Policy feature in Kubernetes lets you define rules for ingress and egress traffic between pods in a cluster.
Azure provides two ways to implement network policy. You choose a network policy option when you create an AKS cluster. The policy option can't be changed after the cluster is created:
- Azure's own implementation, called Azure Network Policies. The Azure network policy requires the Azure CNI.
- Calico Network Policies, an open-source network and network security solution founded by [Tigera](https://www.tigera.io/).
You can also choose to have no network policy.
For more information about the differences between Azure and Calico network policies and their capabilities, see the [AKS documentation.](https://docs.microsoft.com/en-us/azure/aks/use-network-policies#differences-between-azure-and-calico-policies-and-their-capabilities)
### DNS Prefix
Enter a unique DNS prefix for your cluster's Kubernetes API server FQDN.
### Network Plugin
There are two network plugins: kubenet and Azure CNI.
The [kubenet](https://kubernetes.io/docs/concepts/cluster-administration/network-plugins/#kubenet) Kubernetes plugin is the default configuration for AKS cluster creation. When kubenet is used, each node in the cluster receives a routable IP address. The pods use NAT to communicate with other resources outside the AKS cluster. This approach reduces the number of IP addresses you need to reserve in your network space for pods to use.
With the Azure CNI (advanced) networking plugin, pods get full virtual network connectivity and can be directly reached via their private IP address from connected networks. This plugin requires more IP address space.
For more information on the differences between kubenet and Azure CNI, see the [AKS documentation.](https://docs.microsoft.com/en-us/azure/aks/concepts-network#compare-network-models)
### HTTP Application Routing
When enabled, the HTTP application routing add-on makes it easier to access applications deployed to the AKS cluster. It deploys two components: a [Kubernetes Ingress controller](https://kubernetes.io/docs/concepts/services-networking/ingress/) and an [External-DNS](https://github.com/kubernetes-incubator/external-dns) controller.
For more information, see the [AKS documentation.](https://docs.microsoft.com/en-us/azure/aks/http-application-routing)
### Set Authorized IP Ranges
You can secure access to the Kubernetes API server using [authorized IP address ranges.](https://docs.microsoft.com/en-us/azure/aks/api-server-authorized-ip-ranges#overview-of-api-server-authorized-ip-ranges)
The Kubernetes API server exposes the Kubernetes API. This component provides the interaction for management tools, such as kubectl. AKS provides a single-tenant cluster control plane with a dedicated API server. By default, the API server is assigned a public IP address, and you should control access to it using Kubernetes-based or Azure-based RBAC.
To secure access to the otherwise publicly accessible AKS control plane and API server, you can enable and use authorized IP ranges. These authorized IP ranges only allow defined IP address ranges to communicate with the API server.
However, even if you use authorized IP address ranges, you should still use Kubernetes RBAC or Azure RBAC to authorize users and the actions they request.
### Container Monitoring
Container monitoring gives you performance visibility by collecting memory and processor metrics from controllers, nodes, and containers that are available in Kubernetes through the Metrics API. Container logs are also collected. After you enable monitoring, metrics and logs are automatically collected for you through a containerized version of the Log Analytics agent for Linux. Metrics are written to the metrics store and log data is written to the logs store associated with your [Log Analytics](https://docs.microsoft.com/en-us/azure/azure-monitor/logs/log-query-overview) workspace.
### Log Analytics Workspace Resource Group
The [resource group](https://docs.microsoft.com/en-us/azure/azure-resource-manager/management/overview#resource-groups) containing the Log Analytics Workspace. You must create at least one workspace to use Azure Monitor Logs.
### Log Analytics Workspace Name
Data collected by Azure Monitor Logs is stored in one or more [Log Analytics workspaces.](https://docs.microsoft.com/en-us/azure/azure-monitor/logs/design-logs-deployment) The workspace defines the geographic location of the data, access rights defining which users can access data, and configuration settings such as the pricing tier and data retention.
You must create at least one workspace to use Azure Monitor Logs. A single workspace may be suffxicient for all of your monitoring data, or may choose to create multiple workspaces depending on your requirements. For example, you might have one workspace for your production data and another for testing.
For more information about Azure Monitor Logs, see the [Azure documentation.](https://docs.microsoft.com/en-us/azure/azure-monitor/logs/data-platform-logs)
### Support Private Kubernetes Service
Typically, AKS worker nodes do not get public IPs, regardless of whether the cluster is private. In a private cluster, the control plane does not have a public endpoint.
Rancher can connect to a private AKS cluster in one of two ways.
The first way to ensure that Rancher is running on the same [NAT](https://docs.microsoft.com/en-us/azure/virtual-network/nat-overview) as the AKS nodes.
The second way is to run a command to register the cluster with Rancher. Once the cluster is provisioned, you can run the displayed command anywhere you can connect to the cluster’s Kubernetes API. This command is displayed in a pop-up when you provision an AKS cluster with a private API endpoint enabled.
:::note
Please be aware that when registering an existing AKS cluster, the cluster might take some time, possibly hours, to appear in the `Cluster To register` dropdown list. This outcome will be based on region.
:::
For more information about connecting to an AKS private cluster, see the [AKS documentation.](https://docs.microsoft.com/en-us/azure/aks/private-clusters#options-for-connecting-to-the-private-cluster)
# Node Pools
### Mode
The Azure interface allows users to specify whether a Primary Node Pool relies on either `system` (normally used for control planes) or `user` (what is most typically needed for Rancher).
For Primary Node Pools, you can specify Mode, OS, Count and Size.
System node pools always require running nodes, so they cannot be scaled below one node. At least one system node pool is required.
For subsequent node pools, the Rancher UI forces the default of User. User node pools allow you to scale to zero nodes. User node pools don't run any part of the Kubernetes controlplane.
AKS doesn't expose the nodes that run the Kubernetes controlplane components.
### Availability Zones
[Availability zones](https://docs.microsoft.com/en-us/azure/availability-zones/az-overview) are unique physical locations within a region. Each zone is made up of one or more data centers equipped with independent power, cooling, and networking.
Not all regions have support for availability zones. For a list of Azure regions with availability zones, see the [Azure documentation.](https://docs.microsoft.com/en-us/azure/availability-zones/az-region#azure-regions-with-availability-zones)
### VM Size
Choose a size for each VM in the node pool. For details about each VM size, see [this page.](https://azure.microsoft.com/en-us/pricing/details/virtual-machines/linux/)
### OS Disk Type
The nodes in the node pool can have either managed or ephemeral disks.
[Ephemeral OS disks](https://docs.microsoft.com/en-us/azure/virtual-machines/ephemeral-os-disks) are created on the local virtual machine storage and not saved to the remote Azure Storage. Ephemeral OS disks work well for stateless workloads, where applications are tolerant of individual VM failures, but are more affected by VM deployment time or reimaging the individual VM instances. With Ephemeral OS disk, you get lower read/write latency to the OS disk and faster VM reimage.
[Azure managed disks](https://docs.microsoft.com/en-us/azure/virtual-machines/managed-disks-overview) are block-level storage volumes that are managed by Azure and used with Azure Virtual Machines. Managed disks are designed for 99.999% availability. Managed disks achieve this by providing you with three replicas of your data, allowing for high durability.
### OS Disk Size
The size in GB for the disk for each node.
### Node Count
The number of nodes in the node pool. The maximum number of nodes may be limited by your [Azure subscription.](https://docs.microsoft.com/en-us/azure/azure-resource-manager/management/azure-subscription-service-limits)
### Max Pods Per Node
The maximum number of pods per node defaults to 110 with a maximum of 250.
### Enable Auto Scaling
When auto scaling is enabled, you will need to enter a minimum and maximum node count.
When Auto Scaling is enabled, you can't manually scale the node pool. The scale is controlled by the AKS autoscaler.
docs/reference-guides/cluster-configuration/rancher-server-configuration/eks-cluster-configuration.md
+151
View File
@@ -0,0 +1,151 @@
---
title: EKS Cluster Configuration Reference
shortTitle: EKS Cluster Configuration
weight: 2
---
### Account Access
Complete each drop-down and field using the information obtained for your IAM policy.
| Setting | Description |
| ---------- | -------------------------------------------------------------------------------------------------------------------- |
| Region | From the drop-down choose the geographical region in which to build your cluster. |
| Cloud Credentials | Select the cloud credentials that you created for your IAM policy. For more information on creating cloud credentials in Rancher, refer to [this page.](../../user-settings/manage-cloud-credentials.md) |
### Service Role
Choose a [service role](https://docs.aws.amazon.com/IAM/latest/UserGuide/using-service-linked-roles.html).
Service Role | Description
-------------|---------------------------
Standard: Rancher generated service role | If you choose this role, Rancher automatically adds a service role for use with the cluster.
Custom: Choose from your existing service roles | If you choose this role, Rancher lets you choose from service roles that you're already created within AWS. For more information on creating a custom service role in AWS, see the [Amazon documentation](https://docs.aws.amazon.com/IAM/latest/UserGuide/using-service-linked-roles.html#create-service-linked-role).
### Secrets Encryption
Optional: To encrypt secrets, select or enter a key created in [AWS Key Management Service (KMS)](https://docs.aws.amazon.com/kms/latest/developerguide/overview.html)
### API Server Endpoint Access
Configuring Public/Private API access is an advanced use case. For details, refer to the EKS cluster endpoint access control [documentation.](https://docs.aws.amazon.com/eks/latest/userguide/cluster-endpoint.html)
### Private-only API Endpoints
If you enable private and disable public API endpoint access when creating a cluster, then there is an extra step you must take in order for Rancher to connect to the cluster successfully. In this case, a pop-up will be displayed with a command that you will run on the cluster to register it with Rancher. Once the cluster is provisioned, you can run the displayed command anywhere you can connect to the cluster's Kubernetes API.
There are two ways to avoid this extra manual step:
- You can create the cluster with both private and public API endpoint access on cluster creation. You can disable public access after the cluster is created and in an active state and Rancher will continue to communicate with the EKS cluster.
- You can ensure that Rancher shares a subnet with the EKS cluster. Then security groups can be used to enable Rancher to communicate with the cluster's API endpoint. In this case, the command to register the cluster is not needed, and Rancher will be able to communicate with your cluster. For more information on configuring security groups, refer to the [security groups documentation](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_SecurityGroups.html).
### Public Access Endpoints
Optionally limit access to the public endpoint via explicit CIDR blocks.
If you limit access to specific CIDR blocks, then it is recommended that you also enable the private access to avoid losing network communication to the cluster.
One of the following is required to enable private access:
- Rancher's IP must be part of an allowed CIDR block
- Private access should be enabled, and Rancher must share a subnet with the cluster and have network access to the cluster, which can be configured with a security group
For more information about public and private access to the cluster endpoint, refer to the [Amazon EKS documentation.](https://docs.aws.amazon.com/eks/latest/userguide/cluster-endpoint.html)
### Subnet
| Option | Description |
| ------- | ------------ |
| Standard: Rancher generated VPC and Subnet | While provisioning your cluster, Rancher generates a new VPC with 3 public subnets. |
| Custom: Choose from your existing VPC and Subnets | While provisioning your cluster, Rancher configures your Control Plane and nodes to use a VPC and Subnet that you've already [created in AWS](https://docs.aws.amazon.com/vpc/latest/userguide/what-is-amazon-vpc.html). |
For more information, refer to the AWS documentation for [Cluster VPC Considerations](https://docs.aws.amazon.com/eks/latest/userguide/network_reqs.html). Follow one of the sets of instructions below based on your selection from the previous step.
- [What Is Amazon VPC?](https://docs.aws.amazon.com/vpc/latest/userguide/what-is-amazon-vpc.html)
- [VPCs and Subnets](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Subnets.html)
### Security Group
Amazon Documentation:
- [Cluster Security Group Considerations](https://docs.aws.amazon.com/eks/latest/userguide/sec-group-reqs.html)
- [Security Groups for Your VPC](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_SecurityGroups.html)
- [Create a Security Group](https://docs.aws.amazon.com/vpc/latest/userguide/getting-started-ipv4.html#getting-started-create-security-group)
### Logging
Configure control plane logs to send to Amazon CloudWatch. You are charged the standard CloudWatch Logs data ingestion and storage costs for any logs sent to CloudWatch Logs from your clusters.
Each log type corresponds to a component of the Kubernetes control plane. To learn more about these components, see [Kubernetes Components](https://kubernetes.io/docs/concepts/overview/components/) in the Kubernetes documentation.
For more information on EKS control plane logging, refer to the official [documentation.](https://docs.aws.amazon.com/eks/latest/userguide/control-plane-logs.html)
### Managed Node Groups
Amazon EKS managed node groups automate the provisioning and lifecycle management of nodes (Amazon EC2 instances) for Amazon EKS Kubernetes clusters.
For more information about how node groups work and how they are configured, refer to the [EKS documentation.](https://docs.aws.amazon.com/eks/latest/userguide/managed-node-groups.html)
#### Bring your own launch template
A launch template ID and version can be provided in order to easily configure the EC2 instances in a node group. If a launch template is provided, then none of the settings below will be configurable in Rancher. Therefore, using a launch template would require that all the necessary and desired settings from the list below would need to be specified in the launch template. Also note that if a launch template ID and version is provided, then only the template version can be updated. Using a new template ID would require creating a new managed node group.
| Option | Description | Required/Optional |
| ------ | ----------- | ----------------- |
| Instance Type | Choose the [hardware specs](https://aws.amazon.com/ec2/instance-types/) for the instance you're provisioning. | Required |
| Image ID | Specify a custom AMI for the nodes. Custom AMIs used with EKS must be [configured properly](https://aws.amazon.com/premiumsupport/knowledge-center/eks-custom-linux-ami/) | Optional |
| Node Volume Size | The launch template must specify an EBS volume with the desired size | Required |
| SSH Key | A key to be added to the instances to provide SSH access to the nodes | Optional |
| User Data | Cloud init script in [MIME multi-part format](https://docs.aws.amazon.com/eks/latest/userguide/launch-templates.html#launch-template-user-data) | Optional |
| Instance Resource Tags | Tag each EC2 instance in the node group | Optional |
#### Rancher-managed launch templates
If you do not specify a launch template, then you will be able to configure the above options in the Rancher UI and all of them can be updated after creation. In order to take advantage of all of these options, Rancher will create and manage a launch template for you. Each cluster in Rancher will have one Rancher-managed launch template and each managed node group that does not have a specified launch template will have one version of the managed launch template. The name of this launch template will have the prefix "rancher-managed-lt-" followed by the display name of the cluster. In addition, the Rancher-managed launch template will be tagged with the key "rancher-managed-template" and value "do-not-modify-or-delete" to help identify it as Rancher-managed. It is important that this launch template and its versions not be modified, deleted, or used with any other clusters or managed node groups. Doing so could result in your node groups being "degraded" and needing to be destroyed and recreated.
#### Custom AMIs
If you specify a custom AMI, whether in a launch template or in Rancher, then the image must be [configured properly](https://aws.amazon.com/premiumsupport/knowledge-center/eks-custom-linux-ami/) and you must provide user data to [bootstrap the node](https://docs.aws.amazon.com/eks/latest/userguide/launch-templates.html#launch-template-custom-ami). This is considered an advanced use case and understanding the requirements is imperative.
If you specify a launch template that does not contain a custom AMI, then Amazon will use the [EKS-optimized AMI](https://docs.aws.amazon.com/eks/latest/userguide/eks-optimized-ami.html) for the Kubernetes version and selected region. You can also select a [GPU enabled instance](https://docs.aws.amazon.com/eks/latest/userguide/eks-optimized-ami.html#gpu-ami) for workloads that would benefit from it.
:::note
The GPU enabled instance setting in Rancher is ignored if a custom AMI is provided, either in the dropdown or in a launch template.
:::
#### Spot instances
Spot instances are now [supported by EKS](https://docs.aws.amazon.com/eks/latest/userguide/managed-node-groups.html#managed-node-group-capacity-types-spot). If a launch template is specified, Amazon recommends that the template not provide an instance type. Instead, Amazon recommends providing multiple instance types. If the "Request Spot Instances" checkbox is enabled for a node group, then you will have the opportunity to provide multiple instance types.
:::note
Any selection you made in the instance type dropdown will be ignored in this situation and you must specify at least one instance type to the "Spot Instance Types" section. Furthermore, a launch template used with EKS cannot request spot instances. Requesting spot instances must be part of the EKS configuration.
:::
#### Node Group Settings
The following settings are also configurable. All of these except for the "Node Group Name" are editable after the node group is created.
| Option | Description |
| ------- | ------------ |
| Node Group Name | The name of the node group. |
| Desired ASG Size | The desired number of instances. |
| Maximum ASG Size | The maximum number of instances. This setting won't take effect until the [Cluster Autoscaler](https://docs.aws.amazon.com/eks/latest/userguide/cluster-autoscaler.html) is installed. |
| Minimum ASG Size | The minimum number of instances. This setting won't take effect until the [Cluster Autoscaler](https://docs.aws.amazon.com/eks/latest/userguide/cluster-autoscaler.html) is installed. |
| Labels | Kubernetes labels applied to the nodes in the managed node group. |
| Tags | These are tags for the managed node group and do not propagate to any of the associated resources. |
### Configuring the Refresh Interval
The `eks-refresh-cron` setting is deprecated. It has been migrated to the `eks-refresh` setting, which is an integer representing seconds.
The default value is 300 seconds.
The syncing interval can be changed by running `kubectl edit setting eks-refresh`.
If the `eks-refresh-cron` setting was previously set, the migration will happen automatically.
The shorter the refresh window, the less likely any race conditions will occur, but it does increase the likelihood of encountering request limits that may be in place for AWS APIs.
docs/reference-guides/cluster-configuration/rancher-server-configuration/gke-cluster-configuration/gke-private-clusters.md
+51
View File
@@ -0,0 +1,51 @@
---
title: Private Clusters
weight: 2
---
In GKE, [private clusters](https://cloud.google.com/kubernetes-engine/docs/concepts/private-cluster-concept) are clusters whose nodes are isolated from inbound and outbound traffic by assigning them internal IP addresses only. Private clusters in GKE have the option of exposing the control plane endpoint as a publicly accessible address or as a private address. This is different from other Kubernetes providers, which may refer to clusters with private control plane endpoints as "private clusters" but still allow traffic to and from nodes. You may want to create a cluster with private nodes, with or without a public control plane endpoint, depending on your organization's networking and security requirements. A GKE cluster provisioned from Rancher can use isolated nodes by selecting "Private Cluster" in the Cluster Options (under "Show advanced options"). The control plane endpoint can optionally be made private by selecting "Enable Private Endpoint".
### Private Nodes
Because the nodes in a private cluster only have internal IP addresses, they will not be able to install the cluster agent and Rancher will not be able to fully manage the cluster. This can be overcome in a few ways.
#### Cloud NAT
:::caution
Cloud NAT will [incur charges](https://cloud.google.com/nat/pricing).
:::
If restricting outgoing internet access is not a concern for your organization, use Google's [Cloud NAT](https://cloud.google.com/nat/docs/using-nat) service to allow nodes in the private network to access the internet, enabling them to download the required images from Dockerhub and contact the Rancher management server. This is the simplest solution.
#### Private registry
:::caution
This scenario is not officially supported, but is described for cases in which using the Cloud NAT service is not sufficient.
:::
If restricting both incoming and outgoing traffic to nodes is a requirement, follow the air-gapped installation instructions to set up a private container image [registry](https://rancher.com/docs/rancher/v2.6/en/installation/other-installation-methods/air-gap/) on the VPC where the cluster is going to be, allowing the cluster nodes to access and download the images they need to run the cluster agent. If the control plane endpoint is also private, Rancher will need [direct access](#direct-access) to it.
### Private Control Plane Endpoint
If the cluster has a public endpoint exposed, Rancher will be able to reach the cluster, and no additional steps need to be taken. However, if the cluster has no public endpoint, then considerations must be made to ensure Rancher can access the cluster.
#### Cloud NAT
:::caution
Cloud NAT will [incur charges](https://cloud.google.com/nat/pricing).
:::
As above, if restricting outgoing internet access to the nodes is not a concern, then Google's [Cloud NAT](https://cloud.google.com/nat/docs/using-nat) service can be used to allow the nodes to access the internet. While the cluster is provisioning, Rancher will provide a registration command to run on the cluster. Download the [kubeconfig](https://cloud.google.com/kubernetes-engine/docs/how-to/cluster-access-for-kubectl) for the new cluster and run the provided kubectl command on the cluster. Gaining access
to the cluster in order to run this command can be done by creating a temporary node or using an existing node in the VPC, or by logging on to or creating an SSH tunnel through one of the cluster nodes.
#### Direct access
If the Rancher server is run on the same VPC as the cluster's control plane, it will have direct access to the control plane's private endpoint. The cluster nodes will need to have access to a [private registry](#private-registry) to download images as described above.
You can also use services from Google such as [Cloud VPN](https://cloud.google.com/network-connectivity/docs/vpn/concepts/overview) or [Cloud Interconnect VLAN](https://cloud.google.com/network-connectivity/docs/interconnect) to facilitate connectivity between your organization's network and your Google VPC.
docs/reference-guides/cluster-configuration/rancher-server-configuration/k3s-cluster-configuration.md
+147
View File
@@ -0,0 +1,147 @@
---
title: K3s Cluster Configuration Reference
shortTitle: K3s Cluster Configuration
weight: 6
---
This section covers the configuration options that are available in Rancher for a new or existing K3s Kubernetes cluster.
# Overview
You can configure the Kubernetes options one of two ways:
- [Rancher UI](#configuration-options-in-the-rancher-ui): Use the Rancher UI to select options that are commonly customized when setting up a Kubernetes cluster.
- [Cluster Config File](#cluster-config-file): Instead of using the Rancher UI to choose Kubernetes options for the cluster, advanced users can create a K3s config file. Using a config file allows you to set any of the [options](https://rancher.com/docs/k3s/latest/en/installation/install-options/) available in an K3s installation.
# Configuration Options in the Rancher UI
:::tip
Some advanced configuration options are not exposed in the Rancher UI forms, but they can be enabled by editing the K3s cluster configuration file in YAML. For the complete reference of configurable options for K3s clusters in YAML, see the [K3s documentation.](https://rancher.com/docs/k3s/latest/en/installation/install-options/)
:::
### Basics
#### Kubernetes Version
The version of Kubernetes installed on your cluster nodes. Rancher packages its own version of Kubernetes based on [hyperkube](https://github.com/rancher/hyperkube).
For more detail, see [Upgrading Kubernetes](../../../getting-started/installation-and-upgrade/upgrade-and-roll-back-kubernetes.md).
#### Encrypt Secrets
Option to enable or disable secrets encryption. When enabled, secrets will be encrypted using a AES-CBC key. If disabled, any previously secrets will not be readable until encryption is enabled again. Refer to the [K3s documentation](https://rancher.com/docs/k3s/latest/en/advanced/#secrets-encryption-config-experimental) for details.
#### Project Network Isolation
If your network provider allows project network isolation, you can choose whether to enable or disable inter-project communication.
#### SELinux
Option to enable or disable [SELinux](https://rancher.com/docs/k3s/latest/en/advanced/#selinux-support) support.
#### CoreDNS
By default, [CoreDNS](https://coredns.io/) is installed as the default DNS provider. If CoreDNS is not installed, an alternate DNS provider must be installed yourself. Refer to the [K3s documentation](https://rancher.com/docs/k3s/latest/en/networking/#coredns) for details..
#### Klipper Service LB
Option to enable or disable the [Klipper](https://github.com/rancher/klipper-lb) service load balancer. Refer to the [K3s documentation](https://rancher.com/docs/k3s/latest/en/networking/#service-load-balancer) for details.
#### Traefik Ingress
Option to enable or disable the [Traefik](https://traefik.io/) HTTP reverse proxy and load balancer. For more details and configuration options, see the [K3s documentation](https://rancher.com/docs/k3s/latest/en/networking/#traefik-ingress-controller).
#### Local Storage
Option to enable or disable [local storage](https://rancher.com/docs/k3s/latest/en/storage/) on the node(s).
#### Metrics Server
Option to enable or disable the [metrics server](https://github.com/kubernetes-incubator/metrics-server). If enabled, ensure port 10250 is opened for inbound TCP traffic.
### Add-On Config
Additional Kubernetes manifests, managed as a [Add-on](https://kubernetes.io/docs/concepts/cluster-administration/addons/), to apply to the cluster on startup. Refer to the [K3s documentation](https://rancher.com/docs/k3s/latest/en/helm/#automatically-deploying-manifests-and-helm-charts) for details.
### Agent Environment Vars
Option to set environment variables for [K3s agents](https://rancher.com/docs/k3s/latest/en/architecture/). The environment variables can be set using key value pairs. Refer to the [K3 documentation](https://rancher.com/docs/k3s/latest/en/installation/install-options/agent-config/) for more details.
### etcd
#### Automatic Snapshots
Option to enable or disable recurring etcd snapshots. If enabled, users have the option to configure the frequency of snapshots. For details, refer to the [K3s documentation](https://rancher.com/docs/k3s/latest/en/backup-restore/#creating-snapshots).
#### Metrics
Option to choose whether to expose etcd metrics to the public or only within the cluster.
### Networking
#### Cluster CIDR
IPv4/IPv6 network CIDRs to use for pod IPs (default: 10.42.0.0/16).
#### Service CIDR
IPv4/IPv6 network CIDRs to use for service IPs (default: 10.43.0.0/16).
#### Cluster DNS
IPv4 Cluster IP for coredns service. Should be in your service-cidr range (default: 10.43.0.10).
#### Cluster Domain
Select the domain for the cluster. The default is `cluster.local`.
#### NodePort Service Port Range
Option to change the range of ports that can be used for [NodePort services](https://kubernetes.io/docs/concepts/services-networking/service/#nodeport). The default is `30000-32767`.
#### TLS Alternate Names
Add additional hostnames or IPv4/IPv6 addresses as Subject Alternative Names on the server TLS cert.
#### Authorized Cluster Endpoint
Authorized Cluster Endpoint can be used to directly access the Kubernetes API server, without requiring communication through Rancher.
For more detail on how an authorized cluster endpoint works and why it is used, refer to the [architecture section.](../../../pages-for-subheaders/rancher-manager-architecture.md#4-authorized-cluster-endpoint)
We recommend using a load balancer with the authorized cluster endpoint. For details, refer to the [recommended architecture section.](../../rancher-manager-architecture/architecture-recommendations.md#architecture-for-an-authorized-cluster-endpoint)
### Registries
Select the image repository to pull Rancher images from. For more details and configuration options, see the [K3s documentation](https://rancher.com/docs/k3s/latest/en/installation/private-registry/).
### Upgrade Strategy
#### Controle Plane Concurrency
Select how many nodes can be upgraded at the same time. Can be a fixed number or percentage.
#### Worker Concurrency
Select how many nodes can be upgraded at the same time. Can be a fixed number or percentage.
#### Drain Nodes (Control Plane)
Option to remove all pods from the node prior to upgrading.
#### Drain Nodes (Worker Nodes)
Option to remove all pods from the node prior to upgrading.
### Advanced
Option to set kubelet options for different nodes. For available options, refer to the [Kubernetes documentation](https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/).
# Cluster Config File
Instead of using the Rancher UI forms to choose Kubernetes options for the cluster, advanced users can create an K3s config file. Using a config file allows you to set any of the [options](https://rancher.com/docs/k3s/latest/en/installation/install-options/) available in an K3s installation.
To edit an K3s config file directly from the Rancher UI, click **Edit as YAML**.
docs/reference-guides/cluster-configuration/rancher-server-configuration/rke1-cluster-configuration.md
+378
View File
@@ -0,0 +1,378 @@
---
title: RKE Cluster Configuration Reference
shortTitle: RKE Cluster Configuration
weight: 1
---
When Rancher installs Kubernetes, it uses [RKE](../../../pages-for-subheaders/launch-kubernetes-with-rancher.md) or [RKE2](https://docs.rke2.io/) as the Kubernetes distribution.
This section covers the configuration options that are available in Rancher for a new or existing RKE Kubernetes cluster.
- [Overview](#overview)
- [Editing Clusters with a Form in the Rancher UI](#editing-clusters-with-a-form-in-the-rancher-ui)
- [Editing Clusters with YAML](#editing-clusters-with-yaml)
- [Configuration Options in the Rancher UI](#configuration-options-in-the-rancher-ui)
- [Kubernetes Version](#kubernetes-version)
- [Network Provider](#network-provider)
- [Project Network Isolation](#project-network-isolation)
- [Kubernetes Cloud Providers](#kubernetes-cloud-providers)
- [Private Registries](#private-registries)
- [Authorized Cluster Endpoint](#authorized-cluster-endpoint)
- [Node Pools](#node-pools)
- [NGINX Ingress](#nginx-ingress)
- [Metrics Server Monitoring](#metrics-server-monitoring)
- [Pod Security Policy Support](#pod-security-policy-support)
- [Docker Version on Nodes](#docker-version-on-nodes)
- [Docker Root Directory](#docker-root-directory)
- [Default Pod Security Policy](#default-pod-security-policy)
- [Node Port Range](#node-port-range)
- [Recurring etcd Snapshots](#recurring-etcd-snapshots)
- [Agent Environment Variables](#agent-environment-variables)
- [Updating ingress-nginx](#updating-ingress-nginx)
- [RKE Cluster Config File Reference](#rke-cluster-config-file-reference)
- [Config File Structure in Rancher](#config-file-structure-in-rancher)
- [Default DNS Provider](#default-dns-provider)
- [Rancher Specific Parameters in YAML](#rancher-specific-parameters-in-yaml)
- [docker_root_dir](#docker_root_dir)
- [enable_cluster_monitoring](#enable_cluster_monitoring)
- [enable_network_policy](#enable_network_policy)
- [local_cluster_auth_endpoint](#local_cluster_auth_endpoint)
- [Custom Network Plug-in](#custom-network-plug-in)
# Overview
You can configure the Kubernetes options one of two ways:
- [Rancher UI](#rancher-ui-options): Use the Rancher UI to select options that are commonly customized when setting up a Kubernetes cluster.
- [Cluster Config File](#cluster-config-file): Instead of using the Rancher UI to choose Kubernetes options for the cluster, advanced users can create an RKE config file. Using a config file allows you to set any of the options available in an RKE installation, except for system_images configuration, by specifying them in YAML.
The RKE cluster config options are nested under the `rancher_kubernetes_engine_config` directive. For more information, see the section about the [cluster config file.](#cluster-config-file)
In [clusters launched by RKE](../../../pages-for-subheaders/launch-kubernetes-with-rancher.md), you can edit any of the remaining options that follow.
For an example of RKE config file syntax, see the [RKE documentation](https://rancher.com/docs/rke/latest/en/example-yamls/).
The forms in the Rancher UI don't include all advanced options for configuring RKE. For the complete reference of configurable options for RKE Kubernetes clusters in YAML, see the [RKE documentation.](https://rancher.com/docs/rke/latest/en/config-options/)
# Editing Clusters with a Form in the Rancher UI
To edit your cluster,
1. In the upper left corner, click **☰ > Cluster Management**.
1. Go to the cluster you want to configure and click **⋮ > Edit Config**.
# Editing Clusters with YAML
Instead of using the Rancher UI to choose Kubernetes options for the cluster, advanced users can create an RKE config file. Using a config file allows you to set any of the options available in an RKE installation, except for system_images configuration, by specifying them in YAML.
RKE clusters (also called RKE1 clusters) are edited differently than RKE2 and K3s clusters.
To edit an RKE config file directly from the Rancher UI,
1. Click **☰ > Cluster Management**.
1. Go to the RKE cluster you want to configure. Click and click **⋮ > Edit Config**. This take you to the RKE configuration form. Note: Because cluster provisioning changed in Rancher 2.6, the **⋮ > Edit as YAML** can be used for configuring RKE2 clusters, but it can't be used for editing RKE1 configuration.
1. In the configuration form, scroll down and click **Edit as YAML**.
1. Edit the RKE options under the `rancher_kubernetes_engine_config` directive.
# Configuration Options in the Rancher UI
:::tip
Some advanced configuration options are not exposed in the Rancher UI forms, but they can be enabled by editing the RKE cluster configuration file in YAML. For the complete reference of configurable options for RKE Kubernetes clusters in YAML, see the [RKE documentation.](https://rancher.com/docs/rke/latest/en/config-options/)
:::
### Kubernetes Version
The version of Kubernetes installed on your cluster nodes. Rancher packages its own version of Kubernetes based on [hyperkube](https://github.com/rancher/hyperkube).
For more detail, see [Upgrading Kubernetes](../../../getting-started/installation-and-upgrade/upgrade-and-roll-back-kubernetes.md).
### Network Provider
The [Network Provider](https://kubernetes.io/docs/concepts/cluster-administration/networking/) that the cluster uses. For more details on the different networking providers, please view our [Networking FAQ](../../../faq/container-network-interface-providers.md).
:::caution
After you launch the cluster, you cannot change your network provider. Therefore, choose which network provider you want to use carefully, as Kubernetes doesn't allow switching between network providers. Once a cluster is created with a network provider, changing network providers would require you tear down the entire cluster and all its applications.
:::
Out of the box, Rancher is compatible with the following network providers:
- [Canal](https://github.com/projectcalico/canal)
- [Flannel](https://github.com/coreos/flannel#flannel)
- [Calico](https://docs.projectcalico.org/v3.11/introduction/)
- [Weave](https://github.com/weaveworks/weave)
:::note Notes on Weave:
When Weave is selected as network provider, Rancher will automatically enable encryption by generating a random password. If you want to specify the password manually, please see how to configure your cluster using a [Config File](cluster-provisioning/rke-clusters/options/#cluster-config-file) and the [Weave Network Plug-in Options](https://rancher.com/docs/rke/latest/en/config-options/add-ons/network-plugins/#weave-network-plug-in-options).
:::
### Project Network Isolation
If your network provider allows project network isolation, you can choose whether to enable or disable inter-project communication.
Project network isolation is available if you are using any RKE network plugin that supports the enforcement of Kubernetes network policies, such as Canal or the Cisco ACI plugin.
### Kubernetes Cloud Providers
You can configure a [Kubernetes cloud provider](../../../pages-for-subheaders/set-up-cloud-providers.md). If you want to use dynamically provisioned [volumes and storage](../../../pages-for-subheaders/create-kubernetes-persistent-storage.md) in Kubernetes, typically you must select the specific cloud provider in order to use it. For example, if you want to use Amazon EBS, you would need to select the `aws` cloud provider.
:::note
If the cloud provider you want to use is not listed as an option, you will need to use the [config file option](#cluster-config-file) to configure the cloud provider. Please reference the [RKE cloud provider documentation](https://rancher.com/docs/rke/latest/en/config-options/cloud-providers/) on how to configure the cloud provider.
:::
### Private Registries
The cluster-level private registry configuration is only used for provisioning clusters.
There are two main ways to set up private registries in Rancher: by setting up the [global default registry](../../../how-to-guides/advanced-user-guides/authentication-permissions-and-global-configuration/global-default-private-registry.md) through the **Settings** tab in the global view, and by setting up a private registry in the advanced options in the cluster-level settings. The global default registry is intended to be used for air-gapped setups, for registries that do not require credentials. The cluster-level private registry is intended to be used in all setups in which the private registry requires credentials.
If your private registry requires credentials, you need to pass the credentials to Rancher by editing the cluster options for each cluster that needs to pull images from the registry.
The private registry configuration option tells Rancher where to pull the [system images](https://rancher.com/docs/rke/latest/en/config-options/system-images/) or [addon images](https://rancher.com/docs/rke/latest/en/config-options/add-ons/) that will be used in your cluster.
- **System images** are components needed to maintain the Kubernetes cluster.
- **Add-ons** are used to deploy several cluster components, including network plug-ins, the ingress controller, the DNS provider, or the metrics server.
For more information on setting up a private registry for components applied during the provisioning of the cluster, see the [RKE documentation on private registries](https://rancher.com/docs/rke/latest/en/config-options/private-registries/).
Rancher v2.6 introduced the ability to configure [ECR registries for RKE clusters](https://rancher.com/docs/rke/latest/en/config-options/private-registries/#amazon-elastic-container-registry-ecr-private-registry-setup).
### Authorized Cluster Endpoint
Authorized Cluster Endpoint can be used to directly access the Kubernetes API server, without requiring communication through Rancher.
:::note
The authorized cluster endpoint only works on Rancher-launched Kubernetes clusters. In other words, it only works in clusters where Rancher [used RKE](../../../pages-for-subheaders/rancher-manager-architecture.md#tools-for-provisioning-kubernetes-clusters) to provision the cluster. It is not available for clusters in a hosted Kubernetes provider, such as Amazon's EKS.
:::
This is enabled by default in Rancher-launched Kubernetes clusters, using the IP of the node with the `controlplane` role and the default Kubernetes self signed certificates.
For more detail on how an authorized cluster endpoint works and why it is used, refer to the [architecture section.](../../../pages-for-subheaders/rancher-manager-architecture.md#4-authorized-cluster-endpoint)
We recommend using a load balancer with the authorized cluster endpoint. For details, refer to the [recommended architecture section.](../../rancher-manager-architecture/architecture-recommendations.md#architecture-for-an-authorized-cluster-endpoint)
### Node Pools
For information on using the Rancher UI to set up node pools in an RKE cluster, refer to [this page.](../../../pages-for-subheaders/use-new-nodes-in-an-infra-provider.md)
### NGINX Ingress
If you want to publish your applications in a high-availability configuration, and you're hosting your nodes with a cloud-provider that doesn't have a native load-balancing feature, enable this option to use NGINX Ingress within the cluster.
### Metrics Server Monitoring
Option to enable or disable [Metrics Server](https://rancher.com/docs/rke/latest/en/config-options/add-ons/metrics-server/).
Each cloud provider capable of launching a cluster using RKE can collect metrics and monitor for your cluster nodes. Enable this option to view your node metrics from your cloud provider's portal.
### Pod Security Policy Support
Enables [pod security policies](../../../how-to-guides/advanced-user-guides/authentication-permissions-and-global-configuration/create-pod-security-policies.md) for the cluster. After enabling this option, choose a policy using the **Default Pod Security Policy** drop-down.
You must have an existing Pod Security Policy configured before you can use this option.
### Docker Version on Nodes
Configures whether nodes are allowed to run versions of Docker that Rancher doesn't officially support.
If you choose to require a supported Docker version, Rancher will stop pods from running on nodes that don't have a supported Docker version installed.
For details on which Docker versions were tested with each Rancher version, refer to the [support maintenance terms.](https://rancher.com/support-maintenance-terms/)
### Docker Root Directory
If the nodes you are adding to the cluster have Docker configured with a non-default Docker Root Directory (default is `/var/lib/docker`), specify the correct Docker Root Directory in this option.
### Default Pod Security Policy
If you enable **Pod Security Policy Support**, use this drop-down to choose the pod security policy that's applied to the cluster.
### Node Port Range
Option to change the range of ports that can be used for [NodePort services](https://kubernetes.io/docs/concepts/services-networking/service/#nodeport). Default is `30000-32767`.
### Recurring etcd Snapshots
Option to enable or disable [recurring etcd snapshots](https://rancher.com/docs/rke/latest/en/etcd-snapshots/#etcd-recurring-snapshots).
### Agent Environment Variables
Option to set environment variables for [rancher agents](../../../how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/launch-kubernetes-with-rancher/about-rancher-agents.md). The environment variables can be set using key value pairs. If rancher agent requires use of proxy to communicate with Rancher server, `HTTP_PROXY`, `HTTPS_PROXY` and `NO_PROXY` environment variables can be set using agent environment variables.
### Updating ingress-nginx
Clusters that were created before Kubernetes 1.16 will have an `ingress-nginx` `updateStrategy` of `OnDelete`. Clusters that were created with Kubernetes 1.16 or newer will have `RollingUpdate`.
If the `updateStrategy` of `ingress-nginx` is `OnDelete`, you will need to delete these pods to get the correct version for your deployment.
# RKE Cluster Config File Reference
Instead of using the Rancher UI to choose Kubernetes options for the cluster, advanced users can create an RKE config file. Using a config file allows you to set any of the [options available](https://rancher.com/docs/rke/latest/en/config-options/) in an RKE installation, except for `system_images` configuration. The `system_images` option is not supported when creating a cluster with the Rancher UI or API.
For the complete reference for configurable options for RKE Kubernetes clusters in YAML, see the [RKE documentation.](https://rancher.com/docs/rke/latest/en/config-options/)
### Config File Structure in Rancher
RKE (Rancher Kubernetes Engine) is the tool that Rancher uses to provision Kubernetes clusters. Rancher's cluster config files used to have the same structure as [RKE config files,](https://rancher.com/docs/rke/latest/en/example-yamls/) but the structure changed so that in Rancher, RKE cluster config items are separated from non-RKE config items. Therefore, configuration for your cluster needs to be nested under the `rancher_kubernetes_engine_config` directive in the cluster config file. Cluster config files created with earlier versions of Rancher will need to be updated for this format. An example cluster config file is included below.
<details id="v2.3.0-cluster-config-file">
<summary>Example Cluster Config File</summary>
```yaml
#
# Cluster Config
#
docker_root_dir: /var/lib/docker
enable_cluster_alerting: false
enable_cluster_monitoring: false
enable_network_policy: false
local_cluster_auth_endpoint:
enabled: true
#
# Rancher Config
#
rancher_kubernetes_engine_config: # Your RKE template config goes here.
addon_job_timeout: 30
authentication:
strategy: x509
ignore_docker_version: true
#
# # Currently only nginx ingress provider is supported.
# # To disable ingress controller, set `provider: none`
# # To enable ingress on specific nodes, use the node_selector, eg:
# provider: nginx
# node_selector:
# app: ingress
#
ingress:
provider: nginx
kubernetes_version: v1.15.3-rancher3-1
monitoring:
provider: metrics-server
#
# If you are using calico on AWS
#
# network:
# plugin: calico
# calico_network_provider:
# cloud_provider: aws
#
# # To specify flannel interface
#
# network:
# plugin: flannel
# flannel_network_provider:
# iface: eth1
#
# # To specify flannel interface for canal plugin
#
# network:
# plugin: canal
# canal_network_provider:
# iface: eth1
#
network:
options:
flannel_backend_type: vxlan
plugin: canal
#
# services:
# kube-api:
# service_cluster_ip_range: 10.43.0.0/16
# kube-controller:
# cluster_cidr: 10.42.0.0/16
# service_cluster_ip_range: 10.43.0.0/16
# kubelet:
# cluster_domain: cluster.local
# cluster_dns_server: 10.43.0.10
#
services:
etcd:
backup_config:
enabled: true
interval_hours: 12
retention: 6
safe_timestamp: false
creation: 12h
extra_args:
election-timeout: 5000
heartbeat-interval: 500
gid: 0
retention: 72h
snapshot: false
uid: 0
kube_api:
always_pull_images: false
pod_security_policy: false
service_node_port_range: 30000-32767
ssh_agent_auth: false
windows_prefered_cluster: false
```
</details>
### Default DNS provider
The table below indicates what DNS provider is deployed by default. See [RKE documentation on DNS provider](https://rancher.com/docs/rke/latest/en/config-options/add-ons/dns/) for more information how to configure a different DNS provider. CoreDNS can only be used on Kubernetes v1.12.0 and higher.
| Rancher version | Kubernetes version | Default DNS provider |
|-------------|--------------------|----------------------|
| v2.2.5 and higher | v1.14.0 and higher | CoreDNS |
| v2.2.5 and higher | v1.13.x and lower | kube-dns |
| v2.2.4 and lower | any | kube-dns |
# Rancher Specific Parameters in YAML
Besides the RKE config file options, there are also Rancher specific settings that can be configured in the Config File (YAML):
### docker_root_dir
See [Docker Root Directory](#docker-root-directory).
### enable_cluster_monitoring
Option to enable or disable [Cluster Monitoring](../../../pages-for-subheaders/monitoring-and-alerting.md).
### enable_network_policy
Option to enable or disable Project Network Isolation.
Project network isolation is available if you are using any RKE network plugin that supports the enforcement of Kubernetes network policies, such as Canal or the Cisco ACI plugin.
### local_cluster_auth_endpoint
See [Authorized Cluster Endpoint](#authorized-cluster-endpoint).
Example:
```yaml
local_cluster_auth_endpoint:
enabled: true
fqdn: "FQDN"
ca_certs: |-
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
```
### Custom Network Plug-in
You can add a custom network plug-in by using the [user-defined add-on functionality](https://rancher.com/docs/rke/latest/en/config-options/add-ons/user-defined-add-ons/) of RKE. You define any add-on that you want deployed after the Kubernetes cluster is deployed.
There are two ways that you can specify an add-on:
- [In-line Add-ons](https://rancher.com/docs/rke/latest/en/config-options/add-ons/user-defined-add-ons/#in-line-add-ons)
- [Referencing YAML Files for Add-ons](https://rancher.com/docs/rke/latest/en/config-options/add-ons/user-defined-add-ons/#referencing-yaml-files-for-add-ons)
For an example of how to configure a custom network plug-in by editing the `cluster.yml`, refer to the [RKE documentation.](https://rancher.com/docs/rke/latest/en/config-options/add-ons/network-plugins/custom-network-plugin-example)
docs/reference-guides/cluster-configuration/rancher-server-configuration/rke2-cluster-configuration.md
+200
View File
@@ -0,0 +1,200 @@
---
title: RKE2 Cluster Configuration Reference
shortTitle: RKE2 Cluster Configuration
weight: 5
---
This section covers the configuration options that are available in Rancher for a new or existing RKE2 Kubernetes cluster.
# Overview
You can configure the Kubernetes options in one of the two following ways:
- [Rancher UI](#configuration-options-in-the-rancher-ui): Use the Rancher UI to select options that are commonly customized when setting up a Kubernetes cluster.
- [Cluster Config File](#cluster-config-file): Instead of using the Rancher UI to choose Kubernetes options for the cluster, advanced users can create an RKE2 config file. Using a config file allows you to set any of the [options](https://docs.rke2.io/install/install_options/install_options) available in an RKE2 installation.
# Configuration Options in the Rancher UI
:::tip
Some advanced configuration options are not exposed in the Rancher UI forms, but they can be enabled by editing the RKE2 cluster configuration file in YAML. For the complete reference of configurable options for RKE2 Kubernetes clusters in YAML, see the [RKE2 documentation.](https://docs.rke2.io/install/install_options/install_options/)
:::
### Basics
#### Kubernetes Version
The version of Kubernetes installed on your cluster nodes. Rancher packages its own version of Kubernetes based on [hyperkube](https://github.com/rancher/hyperkube).
For more detail, see [Upgrading Kubernetes](../../../getting-started/installation-and-upgrade/upgrade-and-roll-back-kubernetes.md).
#### Container Network Provider
The [Network Provider](https://kubernetes.io/docs/concepts/cluster-administration/networking/) that the cluster uses.
:::caution
After you launch the cluster, you cannot change your network provider. Therefore, choose which network provider you want to use carefully, as Kubernetes doesn't allow switching between network providers. Once a cluster is created with a network provider, changing network providers would require you to tear down the entire cluster and all its applications.
:::
Out of the box, Rancher is compatible with the following network providers:
- [Canal](https://github.com/projectcalico/canal)
- [Cilium](https://cilium.io/)*
- [Calico](https://docs.projectcalico.org/v3.11/introduction/)
- [Multus](https://github.com/k8snetworkplumbingwg/multus-cni)
\* When using [project network isolation](#project-network-isolation) in the [Cilium CNI](../../../faq/container-network-interface-providers.md#cilium), it is possible to enable cross-node ingress routing. Click the [CNI provider docs](../../../faq/container-network-interface-providers.md#ingress-routing-across-nodes-in-cilium) to learn more.
For more details on the different networking providers and how to configure them, please view our [RKE2 documentation](https://docs.rke2.io/install/network_options/).
##### Dual-stack Networking
[Dual-stack](https://docs.rke2.io/install/network_options/#dual-stack-configuration) networking is supported for all CNI providers. To configure RKE2 in dual-stack mode, set valid IPv4/IPv6 CIDRs for your [Cluster CIDR](#cluster-cidr) and/or [Service CIDR](#service-cidr).
###### Additional Configuration {#dual-stack-additional-config}
When using `cilium` or `multus,cilium` as your container network interface provider, ensure the **Enable IPv6 Support** option is also enabled.
#### Cloud Provider
You can configure a [Kubernetes cloud provider](../../../pages-for-subheaders/set-up-cloud-providers.md). If you want to use dynamically provisioned [volumes and storage](../../../pages-for-subheaders/create-kubernetes-persistent-storage.md) in Kubernetes, typically you must select the specific cloud provider in order to use it. For example, if you want to use Amazon EBS, you would need to select the `aws` cloud provider.
:::note
If the cloud provider you want to use is not listed as an option, you will need to use the [config file option](#cluster-config-file) to configure the cloud provider. Please reference [this documentation](https://rancher.com/docs/rke/latest/en/config-options/cloud-providers/) on how to configure the cloud provider.
:::
#### Default Pod Security Policy
Choose the default [pod security policy](../../../how-to-guides/advanced-user-guides/authentication-permissions-and-global-configuration/create-pod-security-policies.md) for the cluster. Please refer to the [RKE2 documentation](https://docs.rke2.io/security/policies/) on the specifications of each available policy.
#### Worker CIS Profile
Select a [CIS benchmark](../../../pages-for-subheaders/cis-scan-guides.md) to validate the system configuration against.
#### Project Network Isolation
If your network provider allows project network isolation, you can choose whether to enable or disable inter-project communication.
Project network isolation is available if you are using any RKE2 network plugin that supports the enforcement of Kubernetes network policies, such as Canal.
#### SELinux
Option to enable or disable [SELinux](https://docs.rke2.io/security/selinux) support.
#### CoreDNS
By default, [CoreDNS](https://coredns.io/) is installed as the default DNS provider. If CoreDNS is not installed, an alternate DNS provider must be installed yourself. Refer to the [RKE2 documentation](https://docs.rke2.io/networking/#coredns) for additional CoreDNS configurations.
#### NGINX Ingress
If you want to publish your applications in a high-availability configuration, and you're hosting your nodes with a cloud-provider that doesn't have a native load-balancing feature, enable this option to use NGINX Ingress within the cluster. Refer to the [RKE2 documentation](https://docs.rke2.io/networking/#nginx-ingress-controller) for additional configuration options.
Refer to the [RKE2 documentation](https://docs.rke2.io/networking/#nginx-ingress-controller) for additional configuration options.
#### Metrics Server
Option to enable or disable [Metrics Server](https://rancher.com/docs/rke/latest/en/config-options/add-ons/metrics-server/).
Each cloud provider capable of launching a cluster using RKE2 can collect metrics and monitor for your cluster nodes. Enable this option to view your node metrics from your cloud provider's portal.
### Add-On Config
Additional Kubernetes manifests, managed as an [Add-on](https://kubernetes.io/docs/concepts/cluster-administration/addons/), to apply to the cluster on startup. Refer to the [RKE2 documentation](https://docs.rke2.io/helm/#automatically-deploying-manifests-and-helm-charts) for details.
### Agent Environment Vars
Option to set environment variables for [Rancher agents](https://rancher.com/docs/rancher/v2.6/en/cluster-provisioning/rke-clusters/rancher-agents/). The environment variables can be set using key value pairs. Refer to the [RKE2 documentation](https://docs.rke2.io/install/install_options/linux_agent_config/) for more details.
### etcd
#### Automatic Snapshots
Option to enable or disable recurring etcd snapshots. If enabled, users have the option to configure the frequency of snapshots. For details, refer to the [RKE2 documentation](https://docs.rke2.io/backup_restore/#creating-snapshots). Note that with RKE2, snapshots are stored on each etcd node. This varies from RKE1 which only stores one snapshot per cluster.
#### Metrics
Option to choose whether to expose etcd metrics to the public or only within the cluster.
### Networking
#### Cluster CIDR
IPv4 and/or IPv6 network CIDRs to use for pod IPs (default: 10.42.0.0/16).
##### Dual-stack Networking
To configure [dual-stack](https://docs.rke2.io/install/network_options/#dual-stack-configuration) mode, enter a valid IPv4/IPv6 CIDR. For example `10.42.0.0/16,2001:cafe:42:0::/56`.
[Additional configuration](#dual-stack-additional-config) is required when using `cilium` or `multus,cilium` as your [container network](#container-network) interface provider.
#### Service CIDR
IPv4/IPv6 network CIDRs to use for service IPs (default: 10.43.0.0/16).
##### Dual-stack Networking
To configure [dual-stack](https://docs.rke2.io/install/network_options/#dual-stack-configuration) mode, enter a valid IPv4/IPv6 CIDR. For example `10.42.0.0/16,2001:cafe:42:0::/56`.
[Additional configuration](#dual-stack-additional-config) is required when using `cilium ` or `multus,cilium` as your [container network](#container-network) interface provider.
#### Cluster DNS
IPv4 Cluster IP for coredns service. Should be in your service-cidr range (default: 10.43.0.10).
#### Cluster Domain
Select the domain for the cluster. The default is `cluster.local`.
#### NodePort Service Port Range
Option to change the range of ports that can be used for [NodePort services](https://kubernetes.io/docs/concepts/services-networking/service/#nodeport). The default is `30000-32767`.
#### TLS Alternate Names
Add additional hostnames or IPv4/IPv6 addresses as Subject Alternative Names on the server TLS cert.
#### Authorized Cluster Endpoint
Authorized Cluster Endpoint can be used to directly access the Kubernetes API server, without requiring communication through Rancher.
This is enabled by default in Rancher-launched Kubernetes clusters, using the IP of the node with the `controlplane` role and the default Kubernetes self signed certificates.
For more detail on how an authorized cluster endpoint works and why it is used, refer to the [architecture section.](../../../pages-for-subheaders/rancher-manager-architecture.md#4-authorized-cluster-endpoint)
We recommend using a load balancer with the authorized cluster endpoint. For details, refer to the [recommended architecture section.](../../rancher-manager-architecture/architecture-recommendations.md#architecture-for-an-authorized-cluster-endpoint)
### Registries
Select the image repository to pull Rancher images from. For more details and configuration options, see the [RKE2 documentation](https://docs.rke2.io/install/containerd_registry_configuration/).
### Upgrade Strategy
#### Control Plane Concurrency
Select how many nodes can be upgraded at the same time. Can be a fixed number or percentage.
#### Worker Concurrency
Select how many nodes can be upgraded at the same time. Can be a fixed number or percentage.
#### Drain Nodes (Control Plane)
Option to remove all pods from the node prior to upgrading.
#### Drain Nodes (Worker Nodes)
Option to remove all pods from the node prior to upgrading.
### Advanced
Option to set kubelet options for different nodes. For available options, refer to the [Kubernetes documentation](https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/).
# Cluster Config File
Instead of using the Rancher UI forms to choose Kubernetes options for the cluster, advanced users can create an RKE2 config file. Using a config file allows you to set any of the [options](https://docs.rke2.io/install/install_options/install_options) available in an RKE2 installation.
To edit an RKE2 config file directly from the Rancher UI, click **Edit as YAML**.
docs/reference-guides/cluster-configuration/rancher-server-configuration/sync-clusters.md
+35
View File
@@ -0,0 +1,35 @@
---
title: Syncing
weight: 10
---
Syncing is the feature for EKS and GKE clusters that causes Rancher to update the clusters' values so they are up to date with their corresponding cluster object in the hosted Kubernetes provider. This enables Rancher to not be the sole owner of a hosted cluster’s state. Its largest limitation is that processing an update from Rancher and another source at the same time or within 5 minutes of one finishing may cause the state from one source to completely overwrite the other.
### How it works
There are two fields on the Rancher Cluster object that must be understood to understand how syncing works:
1. The config object for the cluster, located on the Spec of the Cluster:
* For EKS, the field is called EKSConfig
* For GKE, the field is called GKEConfig
2. The UpstreamSpec object
* For EKS, this is located on the EKSStatus field on the Status of the Cluster.
* For GKE, this is located on the GKEStatus field on the Status of the Cluster.
The struct types that define these objects can be found in their corresponding operator projects:
* [eks-operator](https://github.com/rancher/eks-operator/blob/master/pkg/apis/eks.cattle.io/v1/types.go)
* [gke-operator](https://github.com/rancher/gke-operator/blob/master/pkg/apis/gke.cattle.io/v1/types.go)
All fields with the exception of the cluster name, the location (region or zone), Imported, and the cloud credential reference, are nillable on this Spec object.
The EKSConfig or GKEConfig represents desired state for its non-nil values. Fields that are non-nil in the config object can be thought of as “managed". When a cluster is created in Rancher, all fields are non-nil and therefore “managed”. When a pre-existing cluster is registered in rancher all nillable fields are nil and are not “managed”. Those fields become managed once their value has been changed by Rancher.
UpstreamSpec represents the cluster as it is in the hosted Kubernetes provider and is refreshed on an interval of 5 minutes. After the UpstreamSpec has been refreshed, Rancher checks if the cluster has an update in progress. If it is updating, nothing further is done. If it is not currently updating, any “managed” fields on EKSConfig or GKEConfig are overwritten with their corresponding value from the recently updated UpstreamSpec.
The effective desired state can be thought of as the UpstreamSpec + all non-nil fields in the EKSConfig or GKEConfig. This is what is displayed in the UI.
If Rancher and another source attempt to update a cluster at the same time or within the 5 minute refresh window of an update finishing, then it is likely any “managed” fields can be caught in a race condition. To use EKS as an example, a cluster may have PrivateAccess as a managed field. If PrivateAccess is false and then enabled in EKS console, then finishes at 11:01, and then tags are updated from Rancher before 11:05 the value will likely be overwritten. This would also occur if tags were updated while the cluster was processing the update. If the cluster was registered and the PrivateAccess fields was nil then this issue should not occur in the aforementioned case.
docs/reference-guides/cluster-configuration/rancher-server-configuration/use-existing-nodes/rancher-agent-options.md
+54
View File
@@ -0,0 +1,54 @@
---
title: Rancher Agent Options
weight: 2500
---
Rancher deploys an agent on each node to communicate with the node. This pages describes the options that can be passed to the agent. To use these options, you will need to [create a cluster with custom nodes](../../../../pages-for-subheaders/use-existing-nodes.md) and add the options to the generated `docker run` command when adding a node.
For an overview of how Rancher communicates with downstream clusters using node agents, refer to the [architecture section.](../../../../pages-for-subheaders/rancher-manager-architecture.md#3-node-agents)
## General options
| Parameter | Environment variable | Description |
| ---------- | -------------------- | ----------- |
| `--server` | `CATTLE_SERVER` | The configured Rancher `server-url` setting which the agent connects to |
| `--token` | `CATTLE_TOKEN` | Token that is needed to register the node in Rancher |
| `--ca-checksum` | `CATTLE_CA_CHECKSUM` | The SHA256 checksum of the configured Rancher `cacerts` setting to validate |
| `--node-name` | `CATTLE_NODE_NAME` | Override the hostname that is used to register the node (defaults to `hostname -s`) |
| `--label` | `CATTLE_NODE_LABEL` | Add node labels to the node. For multiple labels, pass additional `--label` options. (`--label key=value`) |
| `--taints` | `CATTLE_NODE_TAINTS` | Add node taints to the node. For multiple taints, pass additional `--taints` options. (`--taints key=value:effect`) |
## Role options
| Parameter | Environment variable | Description |
| ---------- | -------------------- | ----------- |
| `--all-roles` | `ALL=true` | Apply all roles (`etcd`,`controlplane`,`worker`) to the node |
| `--etcd` | `ETCD=true` | Apply the role `etcd` to the node |
| `--controlplane` | `CONTROL=true` | Apply the role `controlplane` to the node |
| `--worker` | `WORKER=true` | Apply the role `worker` to the node |
## IP address options
| Parameter | Environment variable | Description |
| ---------- | -------------------- | ----------- |
| `--address` | `CATTLE_ADDRESS` | The IP address the node will be registered with (defaults to the IP used to reach `8.8.8.8`) |
| `--internal-address` | `CATTLE_INTERNAL_ADDRESS` | The IP address used for inter-host communication on a private network |
### Dynamic IP address options
For automation purposes, you can't have a specific IP address in a command as it has to be generic to be used for every node. For this, we have dynamic IP address options. They are used as a value to the existing IP address options. This is supported for `--address` and `--internal-address`.
| Value | Example | Description |
| ---------- | -------------------- | ----------- |
| Interface name | `--address eth0` | The first configured IP address will be retrieved from the given interface |
| `ipify` | `--address ipify` | Value retrieved from `https://api.ipify.org` will be used |
| `awslocal` | `--address awslocal` | Value retrieved from `http://169.254.169.254/latest/meta-data/local-ipv4` will be used |
| `awspublic` | `--address awspublic` | Value retrieved from `http://169.254.169.254/latest/meta-data/public-ipv4` will be used |
| `doprivate` | `--address doprivate` | Value retrieved from `http://169.254.169.254/metadata/v1/interfaces/private/0/ipv4/address` will be used |
| `dopublic` | `--address dopublic` | Value retrieved from `http://169.254.169.254/metadata/v1/interfaces/public/0/ipv4/address` will be used |
| `azprivate` | `--address azprivate` | Value retrieved from `http://169.254.169.254/metadata/instance/network/interface/0/ipv4/ipAddress/0/privateIpAddress?api-version=2017-08-01&format=text` will be used |
| `azpublic` | `--address azpublic` | Value retrieved from `http://169.254.169.254/metadata/instance/network/interface/0/ipv4/ipAddress/0/publicIpAddress?api-version=2017-08-01&format=text` will be used |
| `gceinternal` | `--address gceinternal` | Value retrieved from `http://metadata.google.internal/computeMetadata/v1/instance/network-interfaces/0/ip` will be used |
| `gceexternal` | `--address gceexternal` | Value retrieved from `http://metadata.google.internal/computeMetadata/v1/instance/network-interfaces/0/access-configs/0/external-ip` will be used |
| `packetlocal` | `--address packetlocal` | Value retrieved from `https://metadata.packet.net/2009-04-04/meta-data/local-ipv4` will be used |
| `packetpublic` | `--address packetlocal` | Value retrieved from `https://metadata.packet.net/2009-04-04/meta-data/public-ipv4` will be used |
docs/reference-guides/configure-openldap/openldap-config-reference.md
+86
View File
@@ -0,0 +1,86 @@
---
title: OpenLDAP Configuration Reference
weight: 2
---
This section is intended to be used as a reference when setting up an OpenLDAP authentication provider in Rancher.
For further details on configuring OpenLDAP, refer to the [official documentation.](https://www.openldap.org/doc/)
> Before you proceed with the configuration, please familiarize yourself with the concepts of [External Authentication Configuration and Principal Users](../../pages-for-subheaders/about-authentication.md#external-authentication-configuration-and-principal-users).
- [Background: OpenLDAP Authentication Flow](#background-openldap-authentication-flow)
- [OpenLDAP server configuration](#openldap-server-configuration)
- [User/group schema configuration](#user-group-schema-configuration)
- [User schema configuration](#user-schema-configuration)
- [Group schema configuration](#group-schema-configuration)
## Background: OpenLDAP Authentication Flow
1. When a user attempts to login with LDAP credentials, Rancher creates an initial bind to the LDAP server using a service account with permissions to search the directory and read user/group attributes.
2. Rancher then searches the directory for the user by using a search filter based on the provided username and configured attribute mappings.
3. Once the user has been found, they are authenticated with another LDAP bind request using the user's DN and provided password.
4. Once authentication succeeded, Rancher then resolves the group memberships both from the membership attribute in the user's object and by performing a group search based on the configured user mapping attribute.
# OpenLDAP Server Configuration
You will need to enter the address, port, and protocol to connect to your OpenLDAP server. `389` is the standard port for insecure traffic, `636` for TLS traffic.
> **Using TLS?**
>
> If the certificate used by the OpenLDAP server is self-signed or not from a recognized certificate authority, make sure have at hand the CA certificate (concatenated with any intermediate certificates) in PEM format. You will have to paste in this certificate during the configuration so that Rancher is able to validate the certificate chain.
If you are in doubt about the correct values to enter in the user/group Search Base configuration fields, consult your LDAP administrator or refer to the section [Identify Search Base and Schema using ldapsearch](../../how-to-guides/advanced-user-guides/authentication-permissions-and-global-configuration/about-authentication/authentication-config/configure-active-directory.md#annex-identify-search-base-and-schema-using-ldapsearch) in the Active Directory authentication documentation.
<figcaption>OpenLDAP Server Parameters</figcaption>
| Parameter | Description |
|:--|:--|
| Hostname | Specify the hostname or IP address of the OpenLDAP server |
| Port | Specify the port at which the OpenLDAP server is listening for connections. Unencrypted LDAP normally uses the standard port of 389, while LDAPS uses port 636.|
| TLS | Check this box to enable LDAP over SSL/TLS (commonly known as LDAPS). You will also need to paste in the CA certificate if the server uses a self-signed/enterprise-signed certificate. |
| Server Connection Timeout | The duration in number of seconds that Rancher waits before considering the server unreachable. |
| Service Account Distinguished Name | Enter the Distinguished Name (DN) of the user that should be used to bind, search and retrieve LDAP entries. |
| Service Account Password | The password for the service account. |
| User Search Base | Enter the Distinguished Name of the node in your directory tree from which to start searching for user objects. All users must be descendents of this base DN. For example: "ou=people,dc=acme,dc=com".|
| Group Search Base | If your groups live under a different node than the one configured under `User Search Base` you will need to provide the Distinguished Name here. Otherwise leave this field empty. For example: "ou=groups,dc=acme,dc=com".|
# User/Group Schema Configuration
If your OpenLDAP directory deviates from the standard OpenLDAP schema, you must complete the **Customize Schema** section to match it.
Note that the attribute mappings configured in this section are used by Rancher to construct search filters and resolve group membership. It is therefore always recommended to verify that the configuration here matches the schema used in your OpenLDAP.
If you are unfamiliar with the user/group schema used in the OpenLDAP server, consult your LDAP administrator or refer to the section [Identify Search Base and Schema using ldapsearch](../../how-to-guides/advanced-user-guides/authentication-permissions-and-global-configuration/about-authentication/authentication-config/configure-active-directory.md#annex-identify-search-base-and-schema-using-ldapsearch) in the Active Directory authentication documentation.
### User Schema Configuration
The table below details the parameters for the user schema configuration.
<figcaption>User Schema Configuration Parameters</figcaption>
| Parameter | Description |
|:--|:--|
| Object Class | The name of the object class used for user objects in your domain. If defined, only specify the name of the object class - *don't* include it in an LDAP wrapper such as &(objectClass=xxxx) |
| Username Attribute | The user attribute whose value is suitable as a display name. |
| Login Attribute | The attribute whose value matches the username part of credentials entered by your users when logging in to Rancher. This is typically `uid`. |
| User Member Attribute | The user attribute containing the Distinguished Name of groups a user is member of. Usually this is one of `memberOf` or `isMemberOf`. |
| Search Attribute | When a user enters text to add users or groups in the UI, Rancher queries the LDAP server and attempts to match users by the attributes provided in this setting. Multiple attributes can be specified by separating them with the pipe ("\|") symbol. |
| User Enabled Attribute | If the schema of your OpenLDAP server supports a user attribute whose value can be evaluated to determine if the account is disabled or locked, enter the name of that attribute. The default OpenLDAP schema does not support this and the field should usually be left empty. |
| Disabled Status Bitmask | This is the value for a disabled/locked user account. The parameter is ignored if `User Enabled Attribute` is empty. |
### Group Schema Configuration
The table below details the parameters for the group schema configuration.
<figcaption>Group Schema Configuration Parameters</figcaption>
| Parameter | Description |
|:--|:--|
| Object Class | The name of the object class used for group entries in your domain. If defined, only specify the name of the object class - *don't* include it in an LDAP wrapper such as &(objectClass=xxxx) |
| Name Attribute | The group attribute whose value is suitable for a display name. |
| Group Member User Attribute | The name of the **user attribute** whose format matches the group members in the `Group Member Mapping Attribute`. |
| Group Member Mapping Attribute | The name of the group attribute containing the members of a group. |
| Search Attribute | Attribute used to construct search filters when adding groups to clusters or projects in the UI. See description of user schema `Search Attribute`. |
| Group DN Attribute | The name of the group attribute whose format matches the values in the user's group membership attribute. See `User Member Attribute`. |
| Nested Group Membership | This settings defines whether Rancher should resolve nested group memberships. Use only if your organization makes use of these nested memberships (ie. you have groups that contain other groups as members). This option is disabled if you are using Shibboleth. |
docs/reference-guides/installation-references/feature-flags.md
+1
View File
@@ -0,0 +1 @@
<!-- PLACEHOLDER -->
docs/reference-guides/installation-references/helm-chart-options.md
+285
View File
@@ -0,0 +1,285 @@
---
title: Rancher Helm Chart Options
weight: 1
aliases:
- /rancher/v2.6/en/installation/resources/chart-options
---
This page is a configuration reference for the Rancher Helm chart.
For help choosing a Helm chart version, refer to [this page.](../../getting-started/installation-and-upgrade/resources/choose-a-rancher-version.md)
For information on enabling experimental features, refer to [this page.](../../pages-for-subheaders/enable-experimental-features.md)
- [Common Options](#common-options)
- [Advanced Options](#advanced-options)
- [API Audit Log](#api-audit-log)
- [Setting Extra Environment Variables](#setting-extra-environment-variables)
- [TLS Settings](#tls-settings)
- [Customizing your Ingress](#customizing-your-ingress)
- [HTTP Proxy](#http-proxy)
- [Additional Trusted CAs](#additional-trusted-cas)
- [Private Registry and Air Gap Installs](#private-registry-and-air-gap-installs)
- [External TLS Termination](#external-tls-termination)
### Common Options
| Option | Default Value | Description |
| ------------------------- | ------------- | ---------------------------------------------------------------------------------- |
| `bootstrapPassword` | " " | `string` - Set the [bootstrap password](#bootstrap-password) for the first admin user. After logging in, the admin will need to reset their password. A randomly generated bootstrap password is used if this value is not set.
| `hostname` | " " | `string` - the Fully Qualified Domain Name for your Rancher Server |
| `ingress.tls.source` | "rancher" | `string` - Where to get the cert for the ingress. - "rancher, letsEncrypt, secret" |
| `letsEncrypt.email` | " " | `string` - Your email address |
| `letsEncrypt.environment` | "production" | `string` - Valid options: "staging, production" |
| `privateCA` | false | `bool` - Set to true if your cert is signed by a private CA |
<br/>
### Advanced Options
| Option | Default Value | Description |
| ------------------------------ | ----------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- |
| `additionalTrustedCAs` | false | `bool` - See [Additional Trusted CAs](#additional-trusted-cas) |
| `addLocal` | "true" | `string` - Have Rancher detect and import the "local" Rancher server cluster. _Note: This option is no longer available in v2.5.0. Consider using the `restrictedAdmin` option to prevent users from modifying the local cluster._ |
| `antiAffinity` | "preferred" | `string` - AntiAffinity rule for Rancher pods - "preferred, required" |
| `auditLog.destination` | "sidecar" | `string` - Stream to sidecar container console or hostPath volume - "sidecar, hostPath" |
| `auditLog.hostPath` | "/var/log/rancher/audit" | `string` - log file destination on host (only applies when `auditLog.destination` is set to `hostPath`) |
| `auditLog.level` | 0 | `int` - set the [API Audit Log](../../getting-started/installation-and-upgrade/advanced-options/advanced-use-cases/enable-api-audit-log.md) level. 0 is off. [0-3] |
| `auditLog.maxAge` | 1 | `int` - maximum number of days to retain old audit log files (only applies when `auditLog.destination` is set to `hostPath`) |
| `auditLog.maxBackup` | 1 | `int` - maximum number of audit log files to retain (only applies when `auditLog.destination` is set to `hostPath`) |
| `auditLog.maxSize` | 100 | `int` - maximum size in megabytes of the audit log file before it gets rotated (only applies when `auditLog.destination` is set to `hostPath`) |
| `busyboxImage` | "busybox" | `string` - Image location for busybox image used to collect audit logs |
| `certmanager.version` | "" | `string` - set cert-manager compatibility |
| `debug` | false | `bool` - set debug flag on rancher server |
| `extraEnv` | [] | `list` - set additional environment variables for Rancher |
| `imagePullSecrets` | [] | `list` - list of names of Secret resource containing private registry credentials |
| `ingress.configurationSnippet` | "" | `string` - Add additional Nginx configuration. Can be used for proxy configuration. |
| `ingress.extraAnnotations` | {} | `map` - additional annotations to customize the ingress |
| `ingress.enabled` | true | When set to false, Helm will not install a Rancher ingress. Set the option to false to deploy your own ingress. |
| `letsEncrypt.ingress.class` | "" | `string` - optional ingress class for the cert-manager acmesolver ingress that responds to the Let's Encrypt ACME challenges. Options: traefik, nginx. | |
| `noProxy` | "127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,.svc,.cluster.local,cattle-system.svc" | `string` - comma separated list of hostnames or ip address not to use the proxy | |
| `proxy` | "" | `string` - HTTP[S] proxy server for Rancher |
| `rancherImage` | "rancher/rancher" | `string` - rancher image source |
| `rancherImagePullPolicy` | "IfNotPresent" | `string` - Override imagePullPolicy for rancher server images - "Always", "Never", "IfNotPresent" |
| `rancherImageTag` | same as chart version | `string` - rancher/rancher image tag |
| `replicas` | 3 | `int` - Number of Rancher server replicas. Setting to -1 will dynamically choose 1, 2, or 3 based on the number of available nodes in the cluster. |
| `resources` | {} | `map` - rancher pod resource requests & limits |
| `restrictedAdmin` | `false` | `bool` - When this option is set to true, the initial Rancher user has restricted access to the local Kubernetes cluster to prevent privilege escalation. For more information, see the section about the [restricted-admin role.](../../how-to-guides/advanced-user-guides/authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/global-permissions.md#restricted-admin) |
| `systemDefaultRegistry` | "" | `string` - private registry to be used for all system container images, e.g., http://registry.example.com/ |
| `tls` | "ingress" | `string` - See [External TLS Termination](#external-tls-termination) for details. - "ingress, external" |
| `useBundledSystemChart` | `false` | `bool` - select to use the system-charts packaged with Rancher server. This option is used for air gapped installations. |
### Bootstrap Password
When Rancher starts for the first time, a password is randomly generated for the first admin user. When the admin first logs in to Rancher, the UI shows commands that can be used to retrieve the bootstrap password. The admin needs to run those commands and log in with the bootstrap password. Then Rancher gives the admin an opportunity to reset the password.
If you want to use a specific bootstrap password instead of a randomly generated one, provide the password.
```plain
--set bootstrapPassword="rancher"
```
The password, whether provided or generated, will be stored in a Kubernetes secret. After Rancher is installed, the UI will show instructions for how to retrieve the password using kubectl:
```
kubectl get secret --namespace cattle-system bootstrap-secret -o go-template='{{ .data.bootstrapPassword|base64decode}}{{ "\n" }}'
```
### API Audit Log
Enabling the [API Audit Log](../../getting-started/installation-and-upgrade/advanced-options/advanced-use-cases/enable-api-audit-log.md)
You can collect this log as you would any container log. Enable [logging](../../pages-for-subheaders/logging.md) for the `System` Project on the Rancher server cluster.
```plain
--set auditLog.level=1
```
By default enabling Audit Logging will create a sidecar container in the Rancher pod. This container (`rancher-audit-log`) will stream the log to `stdout`. You can collect this log as you would any container log. When using the sidecar as the audit log destination, the `hostPath`, `maxAge`, `maxBackups`, and `maxSize` options do not apply. It's advised to use your OS or Docker daemon's log rotation features to control disk space use. Enable [logging](../../pages-for-subheaders/logging.md) for the Rancher server cluster or System Project.
Set the `auditLog.destination` to `hostPath` to forward logs to volume shared with the host system instead of streaming to a sidecar container. When setting the destination to `hostPath` you may want to adjust the other auditLog parameters for log rotation.
### Setting Extra Environment Variables
You can set extra environment variables for Rancher server using `extraEnv`. This list uses the same `name` and `value` keys as the container manifest definitions. Remember to quote the values.
```plain
--set 'extraEnv[0].name=CATTLE_TLS_MIN_VERSION'
--set 'extraEnv[0].value=1.0'
```
### TLS Settings
When you install Rancher inside of a Kubernetes cluster, TLS is offloaded at the cluster's ingress controller. The possible TLS settings depend on the used ingress controller.
See [TLS settings](tls-settings.md) for more information and options.
### Import `local` Cluster
By default Rancher server will detect and import the `local` cluster it's running on. User with access to the `local` cluster will essentially have "root" access to all the clusters managed by Rancher server.
:::caution
If you turn addLocal off, most Rancher v2.5 features won't work, including the EKS provisioner.
:::
If this is a concern in your environment you can set this option to "false" on your initial install.
This option is only effective on the initial Rancher install. See [Issue 16522](https://github.com/rancher/rancher/issues/16522) for more information.
```plain
--set addLocal="false"
```
### Customizing your Ingress
To customize or use a different ingress with Rancher server you can set your own Ingress annotations.
Example on setting a custom certificate issuer:
```plain
--set ingress.extraAnnotations.'cert-manager\.io/cluster-issuer'=issuer-name
```
Example on setting a static proxy header with `ingress.configurationSnippet`. This value is parsed like a template so variables can be used.
```plain
--set ingress.configurationSnippet='more_set_input_headers X-Forwarded-Host {{ .Values.hostname }};'
```
### HTTP Proxy
Rancher requires internet access for some functionality (helm charts). Use `proxy` to set your proxy server.
Add your IP exceptions to the `noProxy` list. Make sure you add the Pod cluster IP range (default: `10.42.0.0/16`), Service cluster IP range (default: `10.43.0.0/16`), the internal cluster domains (default: `.svc,.cluster.local`) and any worker cluster `controlplane` nodes. Rancher supports CIDR notation ranges in this list.
```plain
--set proxy="http://<username>:<password>@<proxy_url>:<proxy_port>/"
--set noProxy="127.0.0.0/8\,10.0.0.0/8\,172.16.0.0/12\,192.168.0.0/16\,.svc\,.cluster.local"
```
### Additional Trusted CAs
If you have private registries, catalogs or a proxy that intercepts certificates, you may need to add additional trusted CAs to Rancher.
```plain
--set additionalTrustedCAs=true
```
Once the Rancher deployment is created, copy your CA certs in pem format into a file named `ca-additional.pem` and use `kubectl` to create the `tls-ca-additional` secret in the `cattle-system` namespace.
```plain
kubectl -n cattle-system create secret generic tls-ca-additional --from-file=ca-additional.pem=./ca-additional.pem
```
### Private Registry and Air Gap Installs
For details on installing Rancher with a private registry, see the [air gap installation docs.](../../pages-for-subheaders/air-gapped-helm-cli-install.md)
# External TLS Termination
We recommend configuring your load balancer as a Layer 4 balancer, forwarding plain 80/tcp and 443/tcp to the Rancher Management cluster nodes. The Ingress Controller on the cluster will redirect http traffic on port 80 to https on port 443.
You may terminate the SSL/TLS on a L7 load balancer external to the Rancher cluster (ingress). Use the `--set tls=external` option and point your load balancer at port http 80 on all of the Rancher cluster nodes. This will expose the Rancher interface on http port 80. Be aware that clients that are allowed to connect directly to the Rancher cluster will not be encrypted. If you choose to do this we recommend that you restrict direct access at the network level to just your load balancer.
:::note
If you are using a Private CA signed certificate, add `--set privateCA=true` and see [Adding TLS Secrets - Using a Private CA Signed Certificate](../../getting-started/installation-and-upgrade/resources/add-tls-secrets.md) to add the CA cert for Rancher.
:::
Your load balancer must support long lived websocket connections and will need to insert proxy headers so Rancher can route links correctly.
### Configuring Ingress for External TLS when Using NGINX v0.25
In NGINX v0.25, the behavior of NGINX has [changed](https://github.com/kubernetes/ingress-nginx/blob/master/Changelog.md#0220) regarding forwarding headers and external TLS termination. Therefore, in the scenario that you are using external TLS termination configuration with NGINX v0.25, you must edit the `cluster.yml` to enable the `use-forwarded-headers` option for ingress:
```yaml
ingress:
provider: nginx
options:
use-forwarded-headers: 'true'
```
### Required Headers
- `Host`
- `X-Forwarded-Proto`
- `X-Forwarded-Port`
- `X-Forwarded-For`
### Recommended Timeouts
- Read Timeout: `1800 seconds`
- Write Timeout: `1800 seconds`
- Connect Timeout: `30 seconds`
### Health Checks
Rancher will respond `200` to health checks on the `/healthz` endpoint.
### Example NGINX config
This NGINX configuration is tested on NGINX 1.14.
:::caution
This NGINX configuration is only an example and may not suit your environment. For complete documentation, see [NGINX Load Balancing - HTTP Load Balancing](https://docs.nginx.com/nginx/admin-guide/load-balancer/http-load-balancer/).
:::
- Replace `IP_NODE1`, `IP_NODE2` and `IP_NODE3` with the IP addresses of the nodes in your cluster.
- Replace both occurrences of `FQDN` to the DNS name for Rancher.
- Replace `/certs/fullchain.pem` and `/certs/privkey.pem` to the location of the server certificate and the server certificate key respectively.
```
worker_processes 4;
worker_rlimit_nofile 40000;
events {
worker_connections 8192;
}
http {
upstream rancher {
server IP_NODE_1:80;
server IP_NODE_2:80;
server IP_NODE_3:80;
}
map $http_upgrade $connection_upgrade {
default Upgrade;
'' close;
}
server {
listen 443 ssl http2;
server_name FQDN;
ssl_certificate /certs/fullchain.pem;
ssl_certificate_key /certs/privkey.pem;
location / {
proxy_set_header Host $host;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_pass http://rancher;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
# This allows the ability for the execute shell window to remain open for up to 15 minutes. Without this parameter, the default is 1 minute and will automatically close.
proxy_read_timeout 900s;
proxy_buffering off;
}
}
server {
listen 80;
server_name FQDN;
return 301 https://$server_name$request_uri;
}
}
```
docs/reference-guides/installation-references/tls-settings.md
+22
View File
@@ -0,0 +1,22 @@
---
title: TLS Settings
weight: 3
---
Changing the default TLS settings depends on the chosen installation method.
# Running Rancher in a highly available Kubernetes cluster
When you install Rancher inside of a Kubernetes cluster, TLS is offloaded at the cluster's ingress controller. The possible TLS settings depend on the used ingress controller:
* nginx-ingress-controller (default for RKE1 and RKE2): [Default TLS Version and Ciphers](https://kubernetes.github.io/ingress-nginx/user-guide/tls/#default-tls-version-and-ciphers).
* traefik (default for K3s): [TLS Options](https://doc.traefik.io/traefik/https/tls/#tls-options).
# Running Rancher in a single Docker container
The default TLS configuration only accepts TLS 1.2 and secure TLS cipher suites. You can change this by setting the following environment variables:
| Parameter | Description | Default | Available options |
|-----|-----|-----|-----|
| `CATTLE_TLS_MIN_VERSION` | Minimum TLS version | `1.2` | `1.0`, `1.1`, `1.2`, `1.3` |
| `CATTLE_TLS_CIPHERS` | Allowed TLS cipher suites | `TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256`,<br/>`TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384`,<br/>`TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305`,<br/>`TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256`,<br/>`TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384`,<br/>`TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305` | See [Golang tls constants](https://golang.org/pkg/crypto/tls/#pkg-constants) |
docs/reference-guides/kubernetes-concepts.md
+76
View File
@@ -0,0 +1,76 @@
---
title: Kubernetes Concepts
weight: 4
---
This page explains concepts related to Kubernetes that are important for understanding how Rancher works. The descriptions below provide a simplified overview of Kubernetes components. For more details, refer to the [official documentation on Kubernetes components.](https://kubernetes.io/docs/concepts/overview/components/)
This section covers the following topics:
- [About Docker](#about-docker)
- [About Kubernetes](#about-kubernetes)
- [What is a Kubernetes Cluster?](#what-is-a-kubernetes-cluster)
- [Roles for Nodes in Kubernetes Clusters](#roles-for-nodes-in-kubernetes-clusters)
- [etcd Nodes](#etcd-nodes)
- [Controlplane Nodes](#controlplane-nodes)
- [Worker Nodes](#worker-nodes)
- [About Helm](#about-helm)
# About Docker
Docker is the container packaging and runtime standard. Developers build container images from Dockerfiles and distribute container images from Docker registries. [Docker Hub](https://hub.docker.com) is the most popular public registry. Many organizations also set up private Docker registries. Docker is primarily used to manage containers on individual nodes.
:::note
Although Rancher 1.6 supported Docker Swarm clustering technology, it is no longer supported in Rancher 2.x due to the success of Kubernetes.
:::
# About Kubernetes
Kubernetes is the container cluster management standard. YAML files specify containers and other resources that form an application. Kubernetes performs functions such as scheduling, scaling, service discovery, health check, secret management, and configuration management.
# What is a Kubernetes Cluster?
A cluster is a group of computers that work together as a single system.
A _Kubernetes Cluster_ is a cluster that uses the [Kubernetes container-orchestration system](https://kubernetes.io/) to deploy, maintain, and scale Docker containers, allowing your organization to automate application operations.
# Roles for Nodes in Kubernetes Clusters
Each computing resource in a Kubernetes cluster is called a _node_. Nodes can be either bare-metal servers or virtual machines. Kubernetes classifies nodes into three types: _etcd_ nodes, _control plane_ nodes, and _worker_ nodes.
A Kubernetes cluster consists of at least one etcd, controlplane, and worker node.
### etcd Nodes
Rancher uses etcd as a data store in both single node and high-availability installations. In Kubernetes, etcd is also a role for nodes that store the cluster state.
The state of a Kubernetes cluster is maintained in [etcd.](https://kubernetes.io/docs/concepts/overview/components/#etcd) The etcd nodes run the etcd database.
The etcd database component is a distributed key-value store used as Kubernetes storage for all cluster data, such as cluster coordination and state management. It is recommended to run etcd on multiple nodes so that there's always a backup available for failover.
Although you can run etcd on just one node, etcd requires a majority of nodes, a quorum, to agree on updates to the cluster state. The cluster should always contain enough healthy etcd nodes to form a quorum. For a cluster with n members, a quorum is (n/2)+1. For any odd-sized cluster, adding one node will always increase the number of nodes necessary for a quorum.
Three etcd nodes is generally sufficient for smaller clusters and five etcd nodes for large clusters.
### Controlplane Nodes
Controlplane nodes run the Kubernetes API server, scheduler, and controller manager. These nodes take care of routine tasks to ensure that your cluster maintains your configuration. Because all cluster data is stored on your etcd nodes, control plane nodes are stateless. You can run control plane on a single node, although three or more nodes are recommended for redundancy. Additionally, a single node can share the control plane and etcd roles.
### Worker Nodes
Each [worker node](https://kubernetes.io/docs/concepts/architecture/nodes/) runs the following:
- **Kubelets:** An agent that monitors the state of the node, ensuring your containers are healthy.
- **Workloads:** The containers and pods that hold your apps, as well as other types of deployments.
Worker nodes also run storage and networking drivers, and ingress controllers when required. You create as many worker nodes as necessary to run your [workloads](../pages-for-subheaders/workloads-and-pods.md).
# About Helm
For high-availability installations of Rancher, Helm is the tool used to install Rancher on a Kubernetes cluster.
Helm is the package management tool of choice for Kubernetes. Helm charts provide templating syntax for Kubernetes YAML manifest documents. With Helm we can create configurable deployments instead of just using static files. For more information about creating your own catalog of deployments, check out the docs at [https://helm.sh/](https://helm.sh).
For more information on service accounts and cluster role binding, refer to the [Kubernetes documentation.](https://kubernetes.io/docs/reference/access-authn-authz/rbac/)
docs/reference-guides/monitoring-v2-configuration/examples.md
+25
View File
@@ -0,0 +1,25 @@
---
title: Examples
weight: 400
---
### ServiceMonitor
An example ServiceMonitor custom resource can be found [here.](https://github.com/prometheus-operator/prometheus-operator/blob/master/example/prometheus-operator-crd/monitoring.coreos.com_servicemonitors.yaml)
### PodMonitor
An example PodMonitor can be found [here.](https://github.com/prometheus-operator/prometheus-operator/blob/master/example/user-guides/getting-started/example-app-pod-monitor.yaml) An example Prometheus resource that refers to it can be found [here.](https://github.com/prometheus-operator/prometheus-operator/blob/master/example/user-guides/getting-started/prometheus-pod-monitor.yaml)
### PrometheusRule
For users who are familiar with Prometheus, a PrometheusRule contains the alerting and recording rules that you would normally place in a [Prometheus rule file](https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/).
For a more fine-grained application of PrometheusRules within your cluster, the ruleSelector field on a Prometheus resource allows you to select which PrometheusRules should be loaded onto Prometheus based on the labels attached to the PrometheusRules resources.
An example PrometheusRule is on [this page.](https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/user-guides/alerting.md)
### Alertmanager Config
For an example configuration, refer to [this section.](../../how-to-guides/advanced-user-guides/monitoring-v2-configuration-guides/advanced-configuration/alertmanager.md#example-alertmanager-config)
docs/reference-guides/monitoring-v2-configuration/helm-chart-options.md
+77
View File
@@ -0,0 +1,77 @@
---
title: Helm Chart Options
weight: 8
---
- [Configuring Resource Limits and Requests](#configuring-resource-limits-and-requests)
- [Trusted CA for Notifiers](#trusted-ca-for-notifiers)
- [Additional Scrape Configurations](#additional-scrape-configurations)
- [Configuring Applications Packaged within Monitoring V2](#configuring-applications-packaged-within-monitoring-v2)
- [Increase the Replicas of Alertmanager](#increase-the-replicas-of-alertmanager)
- [Configuring the Namespace for a Persistent Grafana Dashboard](#configuring-the-namespace-for-a-persistent-grafana-dashboard)
# Configuring Resource Limits and Requests
The resource requests and limits can be configured when installing `rancher-monitoring`.
The default values are in the [values.yaml](https://github.com/rancher/charts/blob/main/charts/rancher-monitoring/values.yaml) in the `rancher-monitoring` Helm chart.
The default values in the table below are the minimum required resource limits and requests.
| Resource Name | Memory Limit | CPU Limit | Memory Request | CPU Request |
| ------------- | ------------ | ----------- | ---------------- | ------------------ |
| alertmanager | 500Mi | 1000m | 100Mi | 100m |
| grafana | 200Mi | 200m | 100Mi | 100m |
| kube-state-metrics subchart | 200Mi | 100m | 130Mi | 100m |
| prometheus-node-exporter subchart | 50Mi | 200m | 30Mi | 100m |
| prometheusOperator | 500Mi | 200m | 100Mi | 100m |
| prometheus | 2500Mi | 1000m | 1750Mi | 750m |
| **Total** | **3950Mi** | **2700m** | **2210Mi** | **1250m** |
At least 50Gi storage is recommended.
# Trusted CA for Notifiers
If you need to add a trusted CA to your notifier, follow these steps:
1. Create the `cattle-monitoring-system` namespace.
1. Add your trusted CA secret to the `cattle-monitoring-system` namespace.
1. Deploy or upgrade the `rancher-monitoring` Helm chart. In the chart options, reference the secret in **Alerting > Additional Secrets**.
**Result:** The default Alertmanager custom resource will have access to your trusted CA.
# Additional Scrape Configurations
If the scrape configuration you want cannot be specified via a ServiceMonitor or PodMonitor at the moment, you can provide an `additionalScrapeConfigSecret` on deploying or upgrading `rancher-monitoring`.
A [scrape_config section](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config) specifies a set of targets and parameters describing how to scrape them. In the general case, one scrape configuration specifies a single job.
An example of where this might be used is with Istio. For more information, see [this section.](../../explanations/integrations-in-rancher/istio/configuration-options/selectors-and-scrape-configurations.md)
# Configuring Applications Packaged within Monitoring v2
We deploy kube-state-metrics and node-exporter with monitoring v2. Node exporter are deployed as DaemonSets. In the monitoring v2 helm chart, in the values.yaml, each of the things are deployed as sub charts.
We also deploy grafana which is not managed by prometheus.
If you look at what the helm chart is doing like in kube-state-metrics, there are plenty more values that you can set that aren’t exposed in the top level chart.
But in the top level chart you can add values that override values that exist in the sub chart.
### Increase the Replicas of Alertmanager
As part of the chart deployment options, you can opt to increase the number of replicas of the Alertmanager deployed onto your cluster. The replicas can all be managed using the same underlying Alertmanager Config Secret. For more information on the Alertmanager Config Secret, refer to [this section.](../../how-to-guides/advanced-user-guides/monitoring-v2-configuration-guides/advanced-configuration/alertmanager.md#multiple-alertmanager-replicas)
### Configuring the Namespace for a Persistent Grafana Dashboard
To specify that you would like Grafana to watch for ConfigMaps across all namespaces, set this value in the `rancher-monitoring` Helm chart:
```
grafana.sidecar.dashboards.searchNamespace=ALL
```
Note that the RBAC roles exposed by the Monitoring chart to add Grafana Dashboards are still restricted to giving permissions for users to add dashboards in the namespace defined in `grafana.dashboards.namespace`, which defaults to `cattle-dashboards`.
docs/reference-guides/monitoring-v2-configuration/receivers.md
+319
View File
@@ -0,0 +1,319 @@
---
title: Receiver Configuration
shortTitle: Receivers
weight: 1
---
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
The [Alertmanager Config](https://prometheus.io/docs/alerting/latest/configuration/#configuration-file) Secret contains the configuration of an Alertmanager instance that sends out notifications based on alerts it receives from Prometheus.
:::note
This section assumes familiarity with how monitoring components work together. For more information about Alertmanager, see [this section.](../../explanations/integrations-in-rancher/monitoring-and-alerting/how-monitoring-works.md#3-how-alertmanager-works)
:::
- [Creating Receivers in the Rancher UI](#creating-receivers-in-the-rancher-ui)
- [Receiver Configuration](#receiver-configuration)
- [Slack](#slack)
- [Email](#email)
- [PagerDuty](#pagerduty)
- [Opsgenie](#opsgenie)
- [Webhook](#webhook)
- [Custom](#custom)
- [Teams](#teams)
- [SMS](#sms)
- [Configuring Multiple Receivers](#configuring-multiple-receivers)
- [Example Alertmanager Config](examples.md#example-alertmanager-config)
- [Example Route Config for CIS Scan Alerts](#example-route-config-for-cis-scan-alerts)
- [Trusted CA for Notifiers](#trusted-ca-for-notifiers)
# Creating Receivers in the Rancher UI
:::note Prerequisites:
- The monitoring application needs to be installed.
- If you configured monitoring with an existing Alertmanager Secret, it must have a format that is supported by Rancher's UI. Otherwise you will only be able to make changes based on modifying the Alertmanager Secret directly. Note: We are continuing to make enhancements to what kinds of Alertmanager Configurations we can support using the Routes and Receivers UI, so please [file an issue](https://github.com/rancher/rancher/issues/new) if you have a request for a feature enhancement.
:::
To create notification receivers in the Rancher UI,
<Tabs>
<TabItem value="Rancher v2.6.5+">
1. Go to the cluster where you want to create receivers. Click **Monitoring -> Alerting -> AlertManagerConfigs**.
1. Ciick **Create**.
1. Click **Add Receiver**.
1. Enter a **Name** for the receiver.
1. Configure one or more providers for the receiver. For help filling out the forms, refer to the configuration options below.
1. Click **Create**.
</TabItem>
<TabItem value="Rancher before v2.6.5">
1. Go to the cluster where you want to create receivers. Click **Monitoring** and click **Receiver**.
2. Enter a name for the receiver.
3. Configure one or more providers for the receiver. For help filling out the forms, refer to the configuration options below.
4. Click **Create**.
</TabItem>
</Tabs>
**Result:** Alerts can be configured to send notifications to the receiver(s).
# Receiver Configuration
The notification integrations are configured with the `receiver`, which is explained in the [Prometheus documentation.](https://prometheus.io/docs/alerting/latest/configuration/#receiver)
### Native vs. Non-native Receivers
By default, AlertManager provides native integration with some receivers, which are listed in [this section.](https://prometheus.io/docs/alerting/latest/configuration/#receiver) All natively supported receivers are configurable through the Rancher UI.
For notification mechanisms not natively supported by AlertManager, integration is achieved using the [webhook receiver.](https://prometheus.io/docs/alerting/latest/configuration/#webhook_config) A list of third-party drivers providing such integrations can be found [here.](https://prometheus.io/docs/operating/integrations/#alertmanager-webhook-receiver) Access to these drivers, and their associated integrations, is provided through the Alerting Drivers app. Once enabled, configuring non-native receivers can also be done through the Rancher UI.
Currently the Rancher Alerting Drivers app provides access to the following integrations:
- Microsoft Teams, based on the [prom2teams](https://github.com/idealista/prom2teams) driver
- SMS, based on the [Sachet](https://github.com/messagebird/sachet) driver
The following types of receivers can be configured in the Rancher UI:
- <a href="#slack">Slack</a>
- <a href="#email">Email</a>
- <a href="#pagerduty">PagerDuty</a>
- <a href="#opsgenie">Opsgenie</a>
- <a href="#webhook">Webhook</a>
- <a href="#custom">Custom</a>
- <a href="#teams">Teams</a>
- <a href="#sms">SMS</a>
The custom receiver option can be used to configure any receiver in YAML that cannot be configured by filling out the other forms in the Rancher UI.
# Slack
| Field | Type | Description |
|------|--------------|------|
| URL | String | Enter your Slack webhook URL. For instructions to create a Slack webhook, see the [Slack documentation.](https://get.slack.help/hc/en-us/articles/115005265063-Incoming-WebHooks-for-Slack) |
| Default Channel | String | Enter the name of the channel that you want to send alert notifications in the following format: `#<channelname>`. |
| Proxy URL | String | Proxy for the webhook notifications. |
| Enable Send Resolved Alerts | Bool | Whether to send a follow-up notification if an alert has been resolved (e.g. [Resolved] High CPU Usage). |
# Email
| Field | Type | Description |
|------|--------------|------|
| Default Recipient Address | String | The email address that will receive notifications. |
| Enable Send Resolved Alerts | Bool | Whether to send a follow-up notification if an alert has been resolved (e.g. [Resolved] High CPU Usage). |
SMTP options:
| Field | Type | Description |
|------|--------------|------|
| Sender | String | Enter an email address available on your SMTP mail server that you want to send the notification from. |
| Host | String | Enter the IP address or hostname for your SMTP server. Example: `smtp.email.com`. |
| Use TLS | Bool | Use TLS for encryption. |
| Username | String | Enter a username to authenticate with the SMTP server. |
| Password | String | Enter a password to authenticate with the SMTP server. |
# PagerDuty
| Field | Type | Description |
|------|------|-------|
| Integration Type | String | `Events API v2` or `Prometheus`. |
| Default Integration Key | String | For instructions to get an integration key, see the [PagerDuty documentation.](https://www.pagerduty.com/docs/guides/prometheus-integration-guide/) |
| Proxy URL | String | Proxy for the PagerDuty notifications. |
| Enable Send Resolved Alerts | Bool | Whether to send a follow-up notification if an alert has been resolved (e.g. [Resolved] High CPU Usage). |
# Opsgenie
| Field | Description |
|------|-------------|
| API Key | For instructions to get an API key, refer to the [Opsgenie documentation.](https://docs.opsgenie.com/docs/api-key-management) |
| Proxy URL | Proxy for the Opsgenie notifications. |
| Enable Send Resolved Alerts | Whether to send a follow-up notification if an alert has been resolved (e.g. [Resolved] High CPU Usage). |
Opsgenie Responders:
| Field | Type | Description |
|-------|------|--------|
| Type | String | Schedule, Team, User, or Escalation. For more information on alert responders, refer to the [Opsgenie documentation.](https://docs.opsgenie.com/docs/alert-recipients-and-teams) |
| Send To | String | Id, Name, or Username of the Opsgenie recipient. |
# Webhook
| Field | Description |
|-------|--------------|
| URL | Webhook URL for the app of your choice. |
| Proxy URL | Proxy for the webhook notification. |
| Enable Send Resolved Alerts | Whether to send a follow-up notification if an alert has been resolved (e.g. [Resolved] High CPU Usage). |
<!-- TODO add info on webhook for teams and sms and link to them -->
# Custom
The YAML provided here will be directly appended to your receiver within the Alertmanager Config Secret.
# Teams
### Enabling the Teams Receiver for Rancher Managed Clusters
The Teams receiver is not a native receiver and must be enabled before it can be used. You can enable the Teams receiver for a Rancher managed cluster by going to the Apps page and installing the rancher-alerting-drivers app with the Teams option selected.
1. In the Rancher UI, go to the cluster where you want to install rancher-alerting-drivers and click **Apps & Marketplace**.
1. Click the **Alerting Drivers** app.
1. Click the **Helm Deploy Options** tab.
1. Select the **Teams** option and click **Install**.
1. Take note of the namespace used as it will be required in a later step.
### Configure the Teams Receiver
The Teams receiver can be configured by updating its ConfigMap. For example, the following is a minimal Teams receiver configuration.
```yaml
[Microsoft Teams]
teams-instance-1: https://your-teams-webhook-url
```
When configuration is complete, add the receiver using the steps in [this section](#creating-receivers-in-the-rancher-ui).
Use the example below as the URL where:
- `ns-1` is replaced with the namespace where the `rancher-alerting-drivers` app is installed
```yaml
url: http://rancher-alerting-drivers-prom2teams.ns-1.svc:8089/v2/teams-instance-1
```
<!-- https://github.com/idealista/prom2teams -->
# SMS
### Enabling the SMS Receiver for Rancher Managed Clusters
The SMS receiver is not a native receiver and must be enabled before it can be used. You can enable the SMS receiver for a Rancher managed cluster by going to the Apps page and installing the rancher-alerting-drivers app with the SMS option selected.
1. In the upper left corner, click **☰ > Cluster Management**.
1. On the **Clusters** page, go to the cluster where you want to install `rancher-alerting-drivers` and click **Explore**.
1. In the left navigation bar, click
1. Click the **Alerting Drivers** app.
1. Click the **Helm Deploy Options** tab
1. Select the **SMS** option and click **Install**.
1. Take note of the namespace used as it will be required in a later step.
### Configure the SMS Receiver
The SMS receiver can be configured by updating its ConfigMap. For example, the following is a minimal SMS receiver configuration.
```yaml
providers:
telegram:
token: 'your-token-from-telegram'
receivers:
- name: 'telegram-receiver-1'
provider: 'telegram'
to:
- '123456789'
```
When configuration is complete, add the receiver using the steps in [this section](#creating-receivers-in-the-rancher-ui).
Use the example below as the name and URL, where:
- the name assigned to the receiver, e.g. `telegram-receiver-1`, must match the name in the `receivers.name` field in the ConfigMap, e.g. `telegram-receiver-1`
- `ns-1` in the URL is replaced with the namespace where the `rancher-alerting-drivers` app is installed
```yaml
name: telegram-receiver-1
url http://rancher-alerting-drivers-sachet.ns-1.svc:9876/alert
```
<!-- https://github.com/messagebird/sachet -->
# Configuring Multiple Receivers
By editing the forms in the Rancher UI, you can set up a Receiver resource with all the information Alertmanager needs to send alerts to your notification system.
It is also possible to send alerts to multiple notification systems. One way is to configure the Receiver using custom YAML, in which case you can add the configuration for multiple notification systems, as long as you are sure that both systems should receive the same messages.
You can also set up multiple receivers by using the `continue` option for a route, so that the alerts sent to a receiver continue being evaluated in the next level of the routing tree, which could contain another receiver.
# Example Alertmanager Configs
### Slack
To set up notifications via Slack, the following Alertmanager Config YAML can be placed into the `alertmanager.yaml` key of the Alertmanager Config Secret, where the `api_url` should be updated to use your Webhook URL from Slack:
```yaml
route:
group_by: ['job']
group_wait: 30s
group_interval: 5m
repeat_interval: 3h
receiver: 'slack-notifications'
receivers:
- name: 'slack-notifications'
slack_configs:
- send_resolved: true
text: '{{ template "slack.rancher.text" . }}'
api_url: <user-provided slack webhook url here>
templates:
- /etc/alertmanager/config/*.tmpl
```
### PagerDuty
To set up notifications via PagerDuty, use the example below from the [PagerDuty documentation](https://www.pagerduty.com/docs/guides/prometheus-integration-guide/) as a guideline. This example sets up a route that captures alerts for a database service and sends them to a receiver linked to a service that will directly notify the DBAs in PagerDuty, while all other alerts will be directed to a default receiver with a different PagerDuty integration key.
The following Alertmanager Config YAML can be placed into the `alertmanager.yaml` key of the Alertmanager Config Secret. The `service_key` should be updated to use your PagerDuty integration key and can be found as per the "Integrating with Global Event Routing" section of the PagerDuty documentation. For the full list of configuration options, refer to the [Prometheus documentation](https://prometheus.io/docs/alerting/latest/configuration/#pagerduty_config).
```yaml
route:
group_by: [cluster]
receiver: 'pagerduty-notifications'
group_interval: 5m
routes:
- match:
service: database
receiver: 'database-notifcations'
receivers:
- name: 'pagerduty-notifications'
pagerduty_configs:
- service_key: 'primary-integration-key'
- name: 'database-notifcations'
pagerduty_configs:
- service_key: 'database-integration-key'
```
# Example Route Config for CIS Scan Alerts
While configuring the routes for `rancher-cis-benchmark` alerts, you can specify the matching using the key-value pair `job: rancher-cis-scan`.
For example, the following example route configuration could be used with a Slack receiver named `test-cis`:
```yaml
spec:
receiver: test-cis
group_by:
# - string
group_wait: 30s
group_interval: 30s
repeat_interval: 30s
match:
job: rancher-cis-scan
# key: string
match_re:
{}
# key: string
```
For more information on enabling alerting for `rancher-cis-benchmark`, see [this section.](../../pages-for-subheaders/cis-scan-guides.md#enabling-alerting-for-rancher-cis-benchmark)
# Trusted CA for Notifiers
If you need to add a trusted CA to your notifier, follow the steps in [this section.](helm-chart-options.md#trusted-ca-for-notifiers)
docs/reference-guides/monitoring-v2-configuration/routes.md
+97
View File
@@ -0,0 +1,97 @@
---
title: Route Configuration
shortTitle: Routes
weight: 5
---
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
The route configuration is the section of the Alertmanager custom resource that controls how the alerts fired by Prometheus are grouped and filtered before they reach the receiver.
When a Route is changed, the Prometheus Operator regenerates the Alertmanager custom resource to reflect the changes.
For more information about configuring routes, refer to the [official Alertmanager documentation.](https://www.prometheus.io/docs/alerting/latest/configuration/#route)
:::note
This section assumes familiarity with how monitoring components work together. For more information, see [this section.](../../explanations/integrations-in-rancher/monitoring-and-alerting/how-monitoring-works.md)
:::
- [Route Restrictions](#route-restrictions)
- [Route Configuration](#route-configuration)
- [Receiver](#receiver)
- [Grouping](#grouping)
- [Matching](#matching)
# Route Restrictions
Alertmanager proxies alerts for Prometheus based on its receivers and a routing tree that filters alerts to certain receivers based on labels.
Alerting drivers proxy alerts for Alertmanager to non-native receivers, such as Microsoft Teams and SMS.
In the Rancher UI for configuring routes and receivers, you can configure routing trees with one root and then a depth of one more level, for a tree with a depth of two. But if you use a `continue` route when configuring Alertmanager directly, you can make the tree deeper.
Each receiver is for one or more notification providers. So if you know that every alert for Slack should also go to PagerDuty, you can configure both in the same receiver.
# Route Configuration
### Note on Labels and Annotations
Labels should be used for identifying information that can affect the routing of notifications. Identifying information about the alert could consist of a container name, or the name of the team that should be notified.
Annotations should be used for information that does not affect who receives the alert, such as a runbook url or error message.
### Receiver
The route needs to refer to a [receiver](#receiver-configuration) that has already been configured.
### Grouping
<Tabs>
<TabItem value="Rancher v2.6.5+">
:::note
As of Rancher v2.6.5, `Group By` now accepts a list of strings instead of key-value pairs. See the [upstream documentation](https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/api.md#route) for details.
:::
| Field | Default | Description |
|-------|--------------|---------|
| Group By | N/a | List of labels to group by. Labels must not be repeated (unique list). Special label "..." (aggregate by all possible labels), if provided, must be the only element in the list. |
| Group Wait | 30s | How long to wait to buffer alerts of the same group before sending initially. |
| Group Interval | 5m | How long to wait before sending an alert that has been added to a group of alerts for which an initial notification has already been sent. |
| Repeat Interval | 4h | How long to wait before re-sending a given alert that has already been sent. |
</TabItem>
<TabItem value="Rancher before v2.6.5">
| Field | Default | Description |
|-------|--------------|---------|
| Group By | N/a | The labels by which incoming alerts are grouped together. For example, `[ group_by: '[' <labelname>, ... ']' ]` Multiple alerts coming in for labels such as `cluster=A` and `alertname=LatencyHigh` can be batched into a single group. To aggregate by all possible labels, use the special value `'...'` as the sole label name, for example: `group_by: ['...']` Grouping by `...` effectively disables aggregation entirely, passing through all alerts as-is. This is unlikely to be what you want, unless you have a very low alert volume or your upstream notification system performs its own grouping. |
| Group Wait | 30s | How long to wait to buffer alerts of the same group before sending initially. |
| Group Interval | 5m | How long to wait before sending an alert that has been added to a group of alerts for which an initial notification has already been sent. |
| Repeat Interval | 4h | How long to wait before re-sending a given alert that has already been sent. |
</TabItem>
</Tabs>
### Matching
The **Match** field refers to a set of equality matchers used to identify which alerts to send to a given Route based on labels defined on that alert. When you add key-value pairs to the Rancher UI, they correspond to the YAML in this format:
```yaml
match:
[ <labelname>: <labelvalue>, ... ]
```
The **Match Regex** field refers to a set of regex-matchers used to identify which alerts to send to a given Route based on labels defined on that alert. When you add key-value pairs in the Rancher UI, they correspond to the YAML in this format:
```yaml
match_re:
[ <labelname>: <regex>, ... ]
```
docs/reference-guides/monitoring-v2-configuration/servicemonitors-and-podmonitors.md
+35
View File
@@ -0,0 +1,35 @@
---
title: ServiceMonitor and PodMonitor Configuration
shortTitle: ServiceMonitors and PodMonitors
weight: 7
---
ServiceMonitors and PodMonitors are both pseudo-CRDs that map the scrape configuration of the Prometheus custom resource.
These configuration objects declaratively specify the endpoints that Prometheus will scrape metrics from.
ServiceMonitors are more commonly used than PodMonitors, and we recommend them for most use cases.
:::note
This section assumes familiarity with how monitoring components work together. For more information about Alertmanager, see [this section.](../../explanations/integrations-in-rancher/monitoring-and-alerting/how-monitoring-works.md)
:::
### ServiceMonitors
This pseudo-CRD maps to a section of the Prometheus custom resource configuration. It declaratively specifies how groups of Kubernetes services should be monitored.
When a ServiceMonitor is created, the Prometheus Operator updates the Prometheus scrape configuration to include the ServiceMonitor configuration. Then Prometheus begins scraping metrics from the endpoint defined in the ServiceMonitor.
Any Services in your cluster that match the labels located within the ServiceMonitor `selector` field will be monitored based on the `endpoints` specified on the ServiceMonitor. For more information on what fields can be specified, please look at the [spec](https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/api.md#servicemonitor) provided by Prometheus Operator.
For more information about how ServiceMonitors work, refer to the [Prometheus Operator documentation.](https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/user-guides/running-exporters.md)
### PodMonitors
This pseudo-CRD maps to a section of the Prometheus custom resource configuration. It declaratively specifies how group of pods should be monitored.
When a PodMonitor is created, the Prometheus Operator updates the Prometheus scrape configuration to include the PodMonitor configuration. Then Prometheus begins scraping metrics from the endpoint defined in the PodMonitor.
Any Pods in your cluster that match the labels located within the PodMonitor `selector` field will be monitored based on the `podMetricsEndpoints` specified on the PodMonitor. For more information on what fields can be specified, please look at the [spec](https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/api.md#podmonitorspec) provided by Prometheus Operator.
docs/reference-guides/pipelines/concepts.md
+36
View File
@@ -0,0 +1,36 @@
---
title: Concepts
weight: 1
---
The purpose of this page is to explain common concepts and terminology related to pipelines.
- **Pipeline:**
A _pipeline_ is a software delivery process that is broken into different stages and steps. Setting up a pipeline can help developers deliver new software as quickly and efficiently as possible. Within Rancher, you can configure pipelines for each of your Rancher projects. A pipeline is based on a specific repository. It defines the process to build, test, and deploy your code. Rancher uses the [pipeline as code](https://jenkins.io/doc/book/pipeline-as-code/) model. Pipeline configuration is represented as a pipeline file in the source code repository, using the file name `.rancher-pipeline.yml` or `.rancher-pipeline.yaml`.
- **Stages:**
A pipeline stage consists of multiple steps. Stages are executed in the order defined in the pipeline file. The steps in a stage are executed concurrently. A stage starts when all steps in the former stage finish without failure.
- **Steps:**
A pipeline step is executed inside a specified stage. A step fails if it exits with a code other than `0`. If a step exits with this failure code, the entire pipeline fails and terminates.
- **Workspace:**
The workspace is the working directory shared by all pipeline steps. In the beginning of a pipeline, source code is checked out to the workspace. The command for every step bootstraps in the workspace. During a pipeline execution, the artifacts from a previous step will be available in future steps. The working directory is an ephemeral volume and will be cleaned out with the executor pod when a pipeline execution is finished.
Typically, pipeline stages include:
- **Build:**
Each time code is checked into your repository, the pipeline automatically clones the repo and builds a new iteration of your software. Throughout this process, the software is typically reviewed by automated tests.
- **Publish:**
After the build is completed, either a Docker image is built and published to a Docker registry or a catalog template is published.
- **Deploy:**
After the artifacts are published, you would release your application so users could start using the updated product.
docs/reference-guides/pipelines/configure-persistent-data.md
+96
View File
@@ -0,0 +1,96 @@
---
title: Configuring Persistent Data for Pipeline Components
weight: 600
---
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
The pipelines' internal Docker registry and the Minio workloads use ephemeral volumes by default. This default storage works out-of-the-box and makes testing easy, but you lose the build images and build logs if the node running the Docker Registry or Minio fails. In most cases this is fine. If you want build images and logs to survive node failures, you can configure the Docker Registry and Minio to use persistent volumes.
This section assumes that you understand how persistent storage works in Kubernetes. For more information, refer to the section on [how storage works.](../../how-to-guides/advanced-user-guides/manage-clusters/create-kubernetes-persistent-storage/manage-persistent-storage/about-persistent-storage.md)
:::note Prerequisites for both parts A and B:
[Persistent volumes](../../pages-for-subheaders/create-kubernetes-persistent-storage.md) must be available for the cluster.
:::
### A. Configuring Persistent Data for Docker Registry
1. Click **☰ > Cluster Management**.
1. Go to the cluster that you created and click **Explore**.
1. Click **Workload**.
1. Find the `docker-registry` workload and select **⋮ > Edit**.
1. Scroll to the **Volumes** section and expand it. Make one of the following selections from the **Add Volume** menu, which is near the bottom of the section:
- **Add Volume > Add a new persistent volume (claim)**
- **Add Volume > Use an existing persistent volume (claim)**
1. Complete the form that displays to choose a persistent volume for the internal Docker registry.
<Tabs>
<TabItem value="Add a new persistent volume">
1. Enter a **Name** for the volume claim.
1. Select a volume claim **Source**:
- If you select **Use a Storage Class to provision a new persistent volume**, select a storage class and enter a **Capacity**.
- If you select **Use an existing persistent volume**, choose a **Persistent Volume** from the drop-down.
1. From the **Customize** section, choose the read/write access for the volume.
1. Click **Define**.
</TabItem>
<TabItem value="Use an existing persistent volume">
1. Enter a **Name** for the volume claim.
1. Choose a **Persistent Volume Claim** from the dropdown.
1. From the **Customize** section, choose the read/write access for the volume.
1. Click **Define**.
</TabItem>
</Tabs>
1. From the **Mount Point** field, enter `/var/lib/registry`, which is the data storage path inside the Docker registry container.
1. Click **Upgrade**.
### B. Configuring Persistent Data for Minio
1. Click **☰ > Cluster Management**.
1. Go to the cluster that you created and click **Explore**.
1. Click **Workload**.
1. Go to the `minio` workload and select **⋮ > Edit**.
1. Scroll to the **Volumes** section and expand it. Make one of the following selections from the **Add Volume** menu, which is near the bottom of the section:
- **Add Volume > Add a new persistent volume (claim)**
- **Add Volume > Use an existing persistent volume (claim)**
1. Complete the form that displays to choose a persistent volume for the internal Docker registry.
<Tabs>
<TabItem value="Add a new persistent volume">
1. Enter a **Name** for the volume claim.
1. Select a volume claim **Source**:
- If you select **Use a Storage Class to provision a new persistent volume**, select a storage class and enter a **Capacity**.
- If you select **Use an existing persistent volume**, choose a **Persistent Volume** from the drop-down.
1. From the **Customize** section, choose the read/write access for the volume.
1. Click **Define**.
</TabItem>
<TabItem value="Use an existing persistent volume">
1. Enter a **Name** for the volume claim.
1. Choose a **Persistent Volume Claim** from the drop-down.
1. From the **Customize** section, choose the read/write access for the volume.
1. Click **Define**.
</TabItem>
</Tabs>
1. From the **Mount Point** field, enter `/data`, which is the data storage path inside the Minio container.
1. Click **Upgrade**.
**Result:** Persistent storage is configured for your pipeline components.
docs/reference-guides/pipelines/example-repositories.md
+90
View File
@@ -0,0 +1,90 @@
---
title: Example Repositories
weight: 500
---
Rancher ships with several example repositories that you can use to familiarize yourself with pipelines. We recommend configuring and testing the example repository that most resembles your environment before using pipelines with your own repositories in a production environment. Use this example repository as a sandbox for repo configuration, build demonstration, etc. Rancher includes example repositories for:
- Go
- Maven
- php
:::note Prerequisites:
- The example repositories are only available if you have not [configured a version control provider](../../how-to-guides/advanced-user-guides/manage-projects/ci-cd-pipelines.md).
- Because the pipelines app was deprecated in favor of Fleet, you will need to turn on the feature flag for legacy features before using pipelines.
- Note that pipelines in Kubernetes 1.21+ are no longer supported.
1. In the upper left corner, click **☰ > Global Settings**.
1. Click **Feature Flags**.
1. Go to the `legacy` feature flag and click **⋮ > Activate**.
:::
To start using these example repositories,
1. [Enable the example repositories](#1-enable-the-example-repositories)
2. [View the example pipeline](#2-view-the-example-pipeline)
3. [Run the example pipeline](#3-run-the-example-pipeline)
### 1. Enable the Example Repositories
By default, the example pipeline repositories are disabled. Enable one (or more) to test out the pipeline feature and see how it works.
1. In the upper left corner, click **☰ > Cluster Management**.
1. Go to the cluster where you want to configure pipelines and click **Explore**.
1. In the dropdown menu in the top navigation bar, select the project where you want to configure pipelines.
1. In the left navigation bar, click **Legacy > Project > Pipelines**.
1. In the **Pipelines** tab, click **Configure Repositories**.
:::note
Example repositories only display if you haven't fetched your own repos.
:::
1. Click **Enable** for one of the example repos (e.g., `https://github.com/rancher/pipeline-example-go.git`). Then click **Done**.
**Results:**
- The example repository is enabled to work with a pipeline is available in the **Pipeline** tab.
- The following workloads are deployed to a new namespace:
- `docker-registry`
- `jenkins`
- `minio`
### 2. View the Example Pipeline
After enabling an example repository, review the pipeline to see how it is set up.
1. In the upper left corner, click **☰ > Cluster Management**.
1. Go to the cluster where you want to configure pipelines and click **Explore**.
1. In the dropdown menu in the top navigation bar, select the project where you want to configure pipelines.
1. In the left navigation bar, click **Legacy > Project > Pipelines**.
1. In the **Pipelines** tab, click **Configure Repositories**.
1. Find the example repository, select **⋮ > Edit Config**. There are two ways to view the pipeline:
* **Rancher UI**: Click on **Edit Config** or **View/Edit YAML** to view the stages and steps of the pipeline. The YAML view shows the `./rancher-pipeline.yml` file.
### 3. Run the Example Pipeline
After enabling an example repository, run the pipeline to see how it works.
1. In the upper left corner, click **☰ > Cluster Management**.
1. Go to the cluster where you want to configure pipelines and click **Explore**.
1. In the dropdown menu in the top navigation bar, select the project where you want to configure pipelines.
1. In the left navigation bar, click **Legacy > Project > Pipelines**.
1. In the **Pipelines** tab, go to the pipeline and select the vertical **⋮ > Run**.
:::note
When you run a pipeline the first time, it takes a few minutes to pull relevant images and provision necessary pipeline components.
:::
**Result:** The pipeline runs. You can see the results in the logs.
### What's Next?
For detailed information about setting up your own pipeline for your repository, [configure a version control provider](../../how-to-guides/advanced-user-guides/manage-projects/ci-cd-pipelines.md), enable a repository and finally configure your pipeline.
docs/reference-guides/pipelines/example-yaml.md
+72
View File
@@ -0,0 +1,72 @@
---
title: Example YAML File
weight: 501
---
Pipelines can be configured either through the UI or using a yaml file in the repository, i.e. `.rancher-pipeline.yml` or `.rancher-pipeline.yaml`.
In the [pipeline configuration reference](pipeline-configuration.md), we provide examples of how to configure each feature using the Rancher UI or using YAML configuration.
Below is a full example `rancher-pipeline.yml` for those who want to jump right in.
```yaml
# example
stages:
- name: Build something
# Conditions for stages
when:
branch: master
event: [ push, pull_request ]
# Multiple steps run concurrently
steps:
- runScriptConfig:
image: busybox
shellScript: echo ${FIRST_KEY} && echo ${ALIAS_ENV}
# Set environment variables in container for the step
env:
FIRST_KEY: VALUE
SECOND_KEY: VALUE2
# Set environment variables from project secrets
envFrom:
- sourceName: my-secret
sourceKey: secret-key
targetKey: ALIAS_ENV
- runScriptConfig:
image: busybox
shellScript: date -R
# Conditions for steps
when:
branch: [ master, dev ]
event: push
- name: Publish my image
steps:
- publishImageConfig:
dockerfilePath: ./Dockerfile
buildContext: .
tag: rancher/rancher:v2.0.0
# Optionally push to remote registry
pushRemote: true
registry: reg.example.com
- name: Deploy some workloads
steps:
- applyYamlConfig:
path: ./deployment.yaml
# branch conditions for the pipeline
branch:
include: [ master, feature/*]
exclude: [ dev ]
# timeout in minutes
timeout: 30
notification:
recipients:
- # Recipient
recipient: "#mychannel"
# ID of Notifier
notifier: "c-wdcsr:n-c9pg7"
- recipient: "test@example.com"
notifier: "c-wdcsr:n-lkrhd"
# Select which statuses you want the notification to be sent
condition: ["Failed", "Success", "Changed"]
# Ability to override the default message (Optional)
message: "my-message"
```
docs/reference-guides/pipelines/pipeline-configuration.md
+660
View File
@@ -0,0 +1,660 @@
---
title: Pipeline Configuration Reference
weight: 1
---
In this section, you'll learn how to configure pipelines.
- [Step Types](#step-types)
- [Step Type: Run Script](#step-type-run-script)
- [Step Type: Build and Publish Images](#step-type-build-and-publish-images)
- [Step Type: Publish Catalog Template](#step-type-publish-catalog-template)
- [Step Type: Deploy YAML](#step-type-deploy-yaml)
- [Step Type: Deploy Catalog App](#step-type-deploy-catalog-app)
- [Notifications](#notifications)
- [Timeouts](#timeouts)
- [Triggers and Trigger Rules](#triggers-and-trigger-rules)
- [Environment Variables](#environment-variables)
- [Secrets](#secrets)
- [Pipeline Variable Substitution Reference](#pipeline-variable-substitution-reference)
- [Global Pipeline Execution Settings](#global-pipeline-execution-settings)
- [Executor Quota](#executor-quota)
- [Resource Quota for Executors](#resource-quota-for-executors)
- [Custom CA](#custom-ca)
- [Persistent Data for Pipeline Components](#persistent-data-for-pipeline-components)
- [Example rancher-pipeline.yml](#example-rancher-pipeline-yml)
# Step Types
Within each stage, you can add as many steps as you'd like. When there are multiple steps in one stage, they run concurrently.
Step types include:
- [Run Script](#step-type-run-script)
- [Build and Publish Images](#step-type-build-and-publish-images)
- [Publish Catalog Template](#step-type-publish-catalog-template)
- [Deploy YAML](#step-type-deploy-yaml)
- [Deploy Catalog App](#step-type-deploy-catalog-app)
<!--
### Clone
The first stage is preserved to be a cloning step that checks out source code from your repo. Rancher handles the cloning of the git repository. This action is equivalent to `git clone <repository_link> <workspace_dir>`.
-->
### Configuring Steps By UI
If you haven't added any stages, click **Configure pipeline for this branch** to configure the pipeline through the UI.
1. Add stages to your pipeline execution by clicking **Add Stage**.
1. Enter a **Name** for each stage of your pipeline.
1. For each stage, you can configure [trigger rules](#triggers-and-trigger-rules) by clicking on **Show Advanced Options**. Note: this can always be updated at a later time.
1. After you've created a stage, start [adding steps](#step-types) by clicking **Add a Step**. You can add multiple steps to each stage.
### Configuring Steps by YAML
For each stage, you can add multiple steps. Read more about each [step type](#step-types) and the advanced options to get all the details on how to configure the YAML. This is only a small example of how to have multiple stages with a singular step in each stage.
```yaml
# example
stages:
- name: Build something
# Conditions for stages
when:
branch: master
event: [ push, pull_request ]
# Multiple steps run concurrently
steps:
- runScriptConfig:
image: busybox
shellScript: date -R
- name: Publish my image
steps:
- publishImageConfig:
dockerfilePath: ./Dockerfile
buildContext: .
tag: rancher/rancher:v2.0.0
# Optionally push to remote registry
pushRemote: true
registry: reg.example.com
```
# Step Type: Run Script
The **Run Script** step executes arbitrary commands in the workspace inside a specified container. You can use it to build, test and do more, given whatever utilities the base image provides. For your convenience, you can use variables to refer to metadata of a pipeline execution. Please refer to the [pipeline variable substitution reference](#pipeline-variable-substitution-reference) for the list of available variables.
### Configuring Script by UI
1. From the **Step Type** drop-down, choose **Run Script** and fill in the form.
1. Click **Add**.
### Configuring Script by YAML
```yaml
# example
stages:
- name: Build something
steps:
- runScriptConfig:
image: golang
shellScript: go build
```
# Step Type: Build and Publish Images
The **Build and Publish Image** step builds and publishes a Docker image. This process requires a Dockerfile in your source code's repository to complete successfully.
The option to publish an image to an insecure registry is not exposed in the UI, but you can specify an environment variable in the YAML that allows you to publish an image insecurely.
### Configuring Building and Publishing Images by UI
1. From the **Step Type** drop-down, choose **Build and Publish**.
1. Fill in the rest of the form. Descriptions for each field are listed below. When you're done, click **Add**.
Field | Description |
---------|----------|
Dockerfile Path | The relative path to the Dockerfile in the source code repo. By default, this path is `./Dockerfile`, which assumes the Dockerfile is in the root directory. You can set it to other paths in different use cases (`./path/to/myDockerfile` for example). |
Image Name | The image name in `name:tag` format. The registry address is not required. For example, to build `example.com/repo/my-image:dev`, enter `repo/my-image:dev`. |
Push image to remote repository | An option to set the registry that publishes the image that's built. To use this option, enable it and choose a registry from the drop-down. If this option is disabled, the image is pushed to the internal registry. |
Build Context <br/><br/> (**Show advanced options**)| By default, the root directory of the source code (`.`). For more details, see the Docker [build command documentation](https://docs.docker.com/engine/reference/commandline/build/).
### Configuring Building and Publishing Images by YAML
You can use specific arguments for Docker daemon and the build. They are not exposed in the UI, but they are available in pipeline YAML format, as indicated in the example below. Available environment variables include:
Variable Name | Description
------------------------|------------------------------------------------------------
PLUGIN_DRY_RUN | Disable docker push
PLUGIN_DEBUG | Docker daemon executes in debug mode
PLUGIN_MIRROR | Docker daemon registry mirror
PLUGIN_INSECURE | Docker daemon allows insecure registries
PLUGIN_BUILD_ARGS | Docker build args, a comma separated list
<br/>
```yaml
# This example shows an environment variable being used
# in the Publish Image step. This variable allows you to
# publish an image to an insecure registry:
stages:
- name: Publish Image
steps:
- publishImageConfig:
dockerfilePath: ./Dockerfile
buildContext: .
tag: repo/app:v1
pushRemote: true
registry: example.com
env:
PLUGIN_INSECURE: "true"
```
# Step Type: Publish Catalog Template
The **Publish Catalog Template** step publishes a version of a catalog app template (i.e. Helm chart) to a git hosted chart repository. It generates a git commit and pushes it to your chart repository. This process requires a chart folder in your source code's repository and a pre-configured secret in the dedicated pipeline namespace to complete successfully. Any variables in the [pipeline variable substitution reference](#pipeline-variable-substitution-reference) is supported for any file in the chart folder.
### Configuring Publishing a Catalog Template by UI
1. From the **Step Type** drop-down, choose **Publish Catalog Template**.
1. Fill in the rest of the form. Descriptions for each field are listed below. When you're done, click **Add**.
Field | Description |
---------|----------|
Chart Folder | The relative path to the chart folder in the source code repo, where the `Chart.yaml` file is located. |
Catalog Template Name | The name of the template. For example, wordpress. |
Catalog Template Version | The version of the template you want to publish, it should be consistent with the version defined in the `Chart.yaml` file. |
Protocol | You can choose to publish via HTTP(S) or SSH protocol. |
Secret | The secret that stores your Git credentials. You need to create a secret in dedicated pipeline namespace in the project before adding this step. If you use HTTP(S) protocol, store Git username and password in `USERNAME` and `PASSWORD` key of the secret. If you use SSH protocol, store Git deploy key in `DEPLOY_KEY` key of the secret. After the secret is created, select it in this option. |
Git URL | The Git URL of the chart repository that the template will be published to. |
Git Branch | The Git branch of the chart repository that the template will be published to. |
Author Name | The author name used in the commit message. |
Author Email | The author email used in the commit message. |
### Configuring Publishing a Catalog Template by YAML
You can add **Publish Catalog Template** steps directly in the `.rancher-pipeline.yml` file.
Under the `steps` section, add a step with `publishCatalogConfig`. You will provide the following information:
* Path: The relative path to the chart folder in the source code repo, where the `Chart.yaml` file is located.
* CatalogTemplate: The name of the template.
* Version: The version of the template you want to publish, it should be consistent with the version defined in the `Chart.yaml` file.
* GitUrl: The git URL of the chart repository that the template will be published to.
* GitBranch: The git branch of the chart repository that the template will be published to.
* GitAuthor: The author name used in the commit message.
* GitEmail: The author email used in the commit message.
* Credentials: You should provide Git credentials by referencing secrets in dedicated pipeline namespace. If you publish via SSH protocol, inject your deploy key to the `DEPLOY_KEY` environment variable. If you publish via HTTP(S) protocol, inject your username and password to `USERNAME` and `PASSWORD` environment variables.
```yaml
# example
stages:
- name: Publish Wordpress Template
steps:
- publishCatalogConfig:
path: ./charts/wordpress/latest
catalogTemplate: wordpress
version: ${CICD_GIT_TAG}
gitUrl: git@github.com:myrepo/charts.git
gitBranch: master
gitAuthor: example-user
gitEmail: user@example.com
envFrom:
- sourceName: publish-keys
sourceKey: DEPLOY_KEY
```
# Step Type: Deploy YAML
This step deploys arbitrary Kubernetes resources to the project. This deployment requires a Kubernetes manifest file to be present in the source code repository. Pipeline variable substitution is supported in the manifest file. You can view an example file at [GitHub](https://github.com/rancher/pipeline-example-go/blob/master/deployment.yaml). Please refer to the [pipeline variable substitution reference](#pipeline-variable-substitution-reference) for the list of available variables.
### Configure Deploying YAML by UI
1. From the **Step Type** drop-down, choose **Deploy YAML** and fill in the form.
1. Enter the **YAML Path**, which is the path to the manifest file in the source code.
1. Click **Add**.
### Configure Deploying YAML by YAML
```yaml
# example
stages:
- name: Deploy
steps:
- applyYamlConfig:
path: ./deployment.yaml
```
# Step Type :Deploy Catalog App
The **Deploy Catalog App** step deploys a catalog app in the project. It will install a new app if it is not present, or upgrade an existing one.
### Configure Deploying Catalog App by UI
1. From the **Step Type** drop-down, choose **Deploy Catalog App**.
1. Fill in the rest of the form. Descriptions for each field are listed below. When you're done, click **Add**.
Field | Description |
---------|----------|
Catalog | The catalog from which the app template will be used. |
Template Name | The name of the app template. For example, wordpress. |
Template Version | The version of the app template you want to deploy. |
Namespace | The target namespace where you want to deploy the app. |
App Name | The name of the app you want to deploy. |
Answers | Key-value pairs of answers used to deploy the app. |
### Configure Deploying Catalog App by YAML
You can add **Deploy Catalog App** steps directly in the `.rancher-pipeline.yml` file.
Under the `steps` section, add a step with `applyAppConfig`. You will provide the following information:
* CatalogTemplate: The ID of the template. This can be found by clicking `Launch app` and selecting `View details` for the app. It is the last part of the URL.
* Version: The version of the template you want to deploy.
* Answers: Key-value pairs of answers used to deploy the app.
* Name: The name of the app you want to deploy.
* TargetNamespace: The target namespace where you want to deploy the app.
```yaml
# example
stages:
- name: Deploy App
steps:
- applyAppConfig:
catalogTemplate: cattle-global-data:library-mysql
version: 0.3.8
answers:
persistence.enabled: "false"
name: testmysql
targetNamespace: test
```
# Timeouts
By default, each pipeline execution has a timeout of 60 minutes. If the pipeline execution cannot complete within its timeout period, the pipeline is aborted.
### Configuring Timeouts by UI
Enter a new value in the **Timeout** field.
### Configuring Timeouts by YAML
In the `timeout` section, enter the timeout value in minutes.
```yaml
# example
stages:
- name: Build something
steps:
- runScriptConfig:
image: busybox
shellScript: ls
# timeout in minutes
timeout: 30
```
# Notifications
You can enable notifications to any notifiers based on the build status of a pipeline. Before enabling notifications, Rancher recommends setting up notifiers so it will be easy to add recipients immediately.
### Configuring Notifications by UI
1. Within the **Notification** section, turn on notifications by clicking **Enable**.
1. Select the conditions for the notification. You can select to get a notification for the following statuses: `Failed`, `Success`, `Changed`. For example, if you want to receive notifications when an execution fails, select **Failed**.
1. If you don't have any existing notifiers, Rancher will provide a warning that no notifiers are set up and provide a link to be able to go to the notifiers page. Follow the [instructions]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/notifiers) to add a notifier. If you already have notifiers, you can add them to the notification by clicking the **Add Recipient** button.
:::note
Notifiers are configured at a cluster level and require a different level of permissions.
:::
1. For each recipient, select which notifier type from the dropdown. Based on the type of notifier, you can use the default recipient or override the recipient with a different one. For example, if you have a notifier for _Slack_, you can update which channel to send the notification to. You can add additional notifiers by clicking **Add Recipient**.
### Configuring Notifications by YAML
In the `notification` section, you will provide the following information:
* **Recipients:** This will be the list of notifiers/recipients that will receive the notification.
* **Notifier:** The ID of the notifier. This can be found by finding the notifier and selecting **View in API** to get the ID.
* **Recipient:** Depending on the type of the notifier, the "default recipient" can be used or you can override this with a different recipient. For example, when configuring a slack notifier, you select a channel as your default recipient, but if you wanted to send notifications to a different channel, you can select a different recipient.
* **Condition:** Select which conditions of when you want the notification to be sent.
* **Message (Optional):** If you want to change the default notification message, you can edit this in the yaml. Note: This option is not available in the UI.
```yaml
# Example
stages:
- name: Build something
steps:
- runScriptConfig:
image: busybox
shellScript: ls
notification:
recipients:
- # Recipient
recipient: "#mychannel"
# ID of Notifier
notifier: "c-wdcsr:n-c9pg7"
- recipient: "test@example.com"
notifier: "c-wdcsr:n-lkrhd"
# Select which statuses you want the notification to be sent
condition: ["Failed", "Success", "Changed"]
# Ability to override the default message (Optional)
message: "my-message"
```
# Triggers and Trigger Rules
After you configure a pipeline, you can trigger it using different methods:
- **Manually:**
After you configure a pipeline, you can trigger a build using the latest CI definition from Rancher UI. When a pipeline execution is triggered, Rancher dynamically provisions a Kubernetes pod to run your CI tasks and then remove it upon completion.
- **Automatically:**
When you enable a repository for a pipeline, webhooks are automatically added to the version control system. When project users interact with the repo by pushing code, opening pull requests, or creating a tag, the version control system sends a webhook to Rancher Server, triggering a pipeline execution.
To use this automation, webhook management permission is required for the repository. Therefore, when users authenticate and fetch their repositories, only those on which they have webhook management permission will be shown.
Trigger rules can be created to have fine-grained control of pipeline executions in your pipeline configuration. Trigger rules come in two types:
- **Run this when:** This type of rule starts the pipeline, stage, or step when a trigger explicitly occurs.
- **Do Not Run this when:** This type of rule skips the pipeline, stage, or step when a trigger explicitly occurs.
If all conditions evaluate to `true`, then the pipeline/stage/step is executed. Otherwise it is skipped. When a pipeline is skipped, none of the pipeline is executed. When a stage/step is skipped, it is considered successful and follow-up stages/steps continue to run.
Wildcard character (`*`) expansion is supported in `branch` conditions.
This section covers the following topics:
- [Configuring pipeline triggers](#configuring-pipeline-triggers)
- [Configuring stage triggers](#configuring-stage-triggers)
- [Configuring step triggers](#configuring-step-triggers)
- [Configuring triggers by YAML](#configuring-triggers-by-yaml)
### Configuring Pipeline Triggers
1. In the upper left corner, click **☰ > Cluster Management**.
1. Go to the cluster where you want to configure pipelines and click **Explore**.
1. In the dropdown menu in the top navigation bar, select the project where you want to configure pipelines.
1. In the left navigation bar, click **Legacy > Project > Pipelines**.
1. From the repository for which you want to manage trigger rules, select the vertical **⋮ > Edit Config**.
1. Click on **Show Advanced Options**.
1. In the **Trigger Rules** section, configure rules to run or skip the pipeline.
1. Click **Add Rule**. In the **Value** field, enter the name of the branch that triggers the pipeline.
1. **Optional:** Add more branches that trigger a build.
1. Click **Done**.
### Configuring Stage Triggers
1. In the upper left corner, click **☰ > Cluster Management**.
1. Go to the cluster where you want to configure pipelines and click **Explore**.
1. In the dropdown menu in the top navigation bar, select the project where you want to configure pipelines.
1. In the left navigation bar, click **Legacy > Project > Pipelines**.
1. From the repository for which you want to manage trigger rules, select the vertical **⋮ > Edit Config**.
1. Find the **stage** that you want to manage trigger rules, click the **Edit** icon for that stage.
1. Click **Show advanced options**.
1. In the **Trigger Rules** section, configure rules to run or skip the stage.
1. Click **Add Rule**.
1. Choose the **Type** that triggers the stage and enter a value.
| Type | Value |
| ------ | -------------------------------------------------------------------- |
| Branch | The name of the branch that triggers the stage. |
| Event | The type of event that triggers the stage. Values are: `Push`, `Pull Request`, `Tag` |
1. Click **Save**.
### Configuring Step Triggers
1. In the upper left corner, click **☰ > Cluster Management**.
1. Go to the cluster where you want to configure pipelines and click **Explore**.
1. In the dropdown menu in the top navigation bar, select the project where you want to configure pipelines.
1. In the left navigation bar, click **Legacy > Project > Pipelines**.
1. From the repository for which you want to manage trigger rules, select the vertical **⋮ > Edit Config**.
1. Find the **step** that you want to manage trigger rules, click the **Edit** icon for that step.
1. Click **Show advanced options**.
1. In the **Trigger Rules** section, configure rules to run or skip the step.
1. Click **Add Rule**.
1. Choose the **Type** that triggers the step and enter a value.
| Type | Value |
| ------ | -------------------------------------------------------------------- |
| Branch | The name of the branch that triggers the step. |
| Event | The type of event that triggers the step. Values are: `Push`, `Pull Request`, `Tag` |
1. Click **Save**.
### Configuring Triggers by YAML
```yaml
# example
stages:
- name: Build something
# Conditions for stages
when:
branch: master
event: [ push, pull_request ]
# Multiple steps run concurrently
steps:
- runScriptConfig:
image: busybox
shellScript: date -R
# Conditions for steps
when:
branch: [ master, dev ]
event: push
# branch conditions for the pipeline
branch:
include: [ master, feature/*]
exclude: [ dev ]
```
# Environment Variables
When configuring a pipeline, certain [step types](#step-types) allow you to use environment variables to configure the step's script.
### Configuring Environment Variables by UI
1. In the upper left corner, click **☰ > Cluster Management**.
1. Go to the cluster where you want to configure pipelines and click **Explore**.
1. In the dropdown menu in the top navigation bar, select the project where you want to configure pipelines.
1. In the left navigation bar, click **Legacy > Project > Pipelines**.
1. From the pipeline for which you want to edit build triggers, select **⋮ > Edit Config**.
1. Within one of the stages, find the **step** that you want to add an environment variable for, click the **Edit** icon.
1. Click **Show advanced options**.
1. Click **Add Variable**, and then enter a key and value in the fields that appear. Add more variables if needed.
1. Add your environment variable(s) into either the script or file.
1. Click **Save**.
### Configuring Environment Variables by YAML
```yaml
# example
stages:
- name: Build something
steps:
- runScriptConfig:
image: busybox
shellScript: echo ${FIRST_KEY} && echo ${SECOND_KEY}
env:
FIRST_KEY: VALUE
SECOND_KEY: VALUE2
```
# Secrets
If you need to use security-sensitive information in your pipeline scripts (like a password), you can pass them in using Kubernetes [secrets](../../how-to-guides/new-user-guides/kubernetes-resources-setup/secrets.md).
### Prerequisite
Create a secret in the same project as your pipeline, or explicitly in the namespace where pipeline build pods run.
<br/>
:::note
Secret injection is disabled on [pull request events](#triggers-and-trigger-rules).
:::
### Configuring Secrets by UI
1. In the upper left corner, click **☰ > Cluster Management**.
1. Go to the cluster where you want to configure pipelines and click **Explore**.
1. In the dropdown menu in the top navigation bar, select the project where you want to configure pipelines.
1. In the left navigation bar, click **Legacy > Project > Pipelines**.
1. From the pipeline for which you want to edit build triggers, select **⋮ > Edit Config**.
1. Within one of the stages, find the **step** that you want to use a secret for, click the **Edit** icon.
1. Click **Show advanced options**.
1. Click **Add From Secret**. Select the secret file that you want to use. Then choose a key. Optionally, you can enter an alias for the key.
1. Click **Save**.
### Configuring Secrets by YAML
```yaml
# example
stages:
- name: Build something
steps:
- runScriptConfig:
image: busybox
shellScript: echo ${ALIAS_ENV}
# environment variables from project secrets
envFrom:
- sourceName: my-secret
sourceKey: secret-key
targetKey: ALIAS_ENV
```
# Pipeline Variable Substitution Reference
For your convenience, the following variables are available for your pipeline configuration scripts. During pipeline executions, these variables are replaced by metadata. You can reference them in the form of `${VAR_NAME}`.
Variable Name | Description
------------------------|------------------------------------------------------------
`CICD_GIT_REPO_NAME` | Repository name (Github organization omitted).
`CICD_GIT_URL` | URL of the Git repository.
`CICD_GIT_COMMIT` | Git commit ID being executed.
`CICD_GIT_BRANCH` | Git branch of this event.
`CICD_GIT_REF` | Git reference specification of this event.
`CICD_GIT_TAG` | Git tag name, set on tag event.
`CICD_EVENT` | Event that triggered the build (`push`, `pull_request` or `tag`).
`CICD_PIPELINE_ID` | Rancher ID for the pipeline.
`CICD_EXECUTION_SEQUENCE` | Build number of the pipeline.
`CICD_EXECUTION_ID` | Combination of `{CICD_PIPELINE_ID}-{CICD_EXECUTION_SEQUENCE}`.
`CICD_REGISTRY` | Address for the Docker registry for the previous publish image step, available in the Kubernetes manifest file of a `Deploy YAML` step.
`CICD_IMAGE` | Name of the image built from the previous publish image step, available in the Kubernetes manifest file of a `Deploy YAML` step. It does not contain the image tag.<br/><br/> [Example](https://github.com/rancher/pipeline-example-go/blob/master/deployment.yaml)
# Global Pipeline Execution Settings
After configuring a version control provider, there are several options that can be configured globally on how pipelines are executed in Rancher.
### Changing Pipeline Settings
:::note Prerequisite:
Because the pipelines app was deprecated in favor of Fleet, you will need to turn on the feature flag for legacy
features before using pipelines. Note that pipelines in Kubernetes 1.21+ are no longer supported.
1. In the upper left corner, click **☰ > Global Settings**.
1. Click **Feature Flags**.
1. Go to the `legacy` feature flag and click **⋮ > Activate**.
:::
To edit these settings:
1. In the upper left corner, click **☰ > Cluster Management**.
1. Go to the cluster where you want to configure pipelines and click **Explore**.
1. In the dropdown menu in the top navigation bar, select the project where you want to configure pipelines.
1. In the left navigation bar, click **Legacy > Project > Pipelines**.
- [Executor Quota](#executor-quota)
- [Resource Quota for Executors](#resource-quota-for-executors)
- [Custom CA](#custom-ca)
### Executor Quota
Select the maximum number of pipeline executors. The _executor quota_ decides how many builds can run simultaneously in the project. If the number of triggered builds exceeds the quota, subsequent builds will queue until a vacancy opens. By default, the quota is `2`. A value of `0` or less removes the quota limit.
### Resource Quota for Executors
Configure compute resources for Jenkins agent containers. When a pipeline execution is triggered, a build pod is dynamically provisioned to run your CI tasks. Under the hood, A build pod consists of one Jenkins agent container and one container for each pipeline step. You can [manage compute resources](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/) for every containers in the pod.
Edit the **Memory Reservation**, **Memory Limit**, **CPU Reservation** or **CPU Limit**, then click **Update Limit and Reservation**.
To configure compute resources for pipeline-step containers:
You can configure compute resources for pipeline-step containers in the `.rancher-pipeline.yml` file.
In a step, you will provide the following information:
* **CPU Reservation (`CpuRequest`)**: CPU request for the container of a pipeline step.
* **CPU Limit (`CpuLimit`)**: CPU limit for the container of a pipeline step.
* **Memory Reservation (`MemoryRequest`)**: Memory request for the container of a pipeline step.
* **Memory Limit (`MemoryLimit`)**: Memory limit for the container of a pipeline step.
```yaml
# example
stages:
- name: Build something
steps:
- runScriptConfig:
image: busybox
shellScript: ls
cpuRequest: 100m
cpuLimit: 1
memoryRequest:100Mi
memoryLimit: 1Gi
- publishImageConfig:
dockerfilePath: ./Dockerfile
buildContext: .
tag: repo/app:v1
cpuRequest: 100m
cpuLimit: 1
memoryRequest:100Mi
memoryLimit: 1Gi
```
:::note
Rancher sets default compute resources for pipeline steps except for `Build and Publish Images` and `Run Script` steps. You can override the default value by specifying compute resources in the same way.
:::
### Custom CA
If you want to use a version control provider with a certificate from a custom/internal CA root, the CA root certificates need to be added as part of the version control provider configuration in order for the pipeline build pods to succeed.
1. Click **Edit cacerts**.
1. Paste in the CA root certificates and click **Save cacerts**.
**Result:** Pipelines can be used and new pods will be able to work with the self-signed-certificate.
# Persistent Data for Pipeline Components
The internal Docker registry and the Minio workloads use ephemeral volumes by default. This default storage works out-of-the-box and makes testing easy, but you lose the build images and build logs if the node running the Docker Registry or Minio fails. In most cases this is fine. If you want build images and logs to survive node failures, you can configure the Docker Registry and Minio to use persistent volumes.
For details on setting up persistent storage for pipelines, refer to [this page.](configure-persistent-data.md)
# Example rancher-pipeline.yml
An example pipeline configuration file is on [this page.](example-yaml.md)
docs/reference-guides/prometheus-federator/rbac.md
+33
View File
@@ -0,0 +1,33 @@
---
title: Role-Based Access Control
shortTitle: RBAC
weight: 2
---
This section describes the expectations for Role-Based Access Control (RBAC) for Prometheus Federator.
As described in the section on [namespaces](monitoring-alerting/prometheus-federator/prometheus-federator#namespaces), Prometheus Federator expects that Project Owners, Project Members, and other users in the cluster with Project-level permissions (e.g. permissions in a certain set of namespaces identified by a single label selector) have minimal permissions in any namespaces except the Project Registration Namespace (which is imported into the project by default) and those that already comprise their projects. Therefore, in order to allow Project Owners to assign specific chart permissions to other users in their Project namespaces, the Helm Project Operator will automatically watch the following bindings:
- ClusterRoleBindings
- RoleBindings in the Project Release Namespace
On observing a change to one of those types of bindings, the Helm Project Operator will check whether the `roleRef` that the the binding points to matches a ClusterRole with the name provided under:
- `helmProjectOperator.releaseRoleBindings.clusterRoleRefs.admin`
- `helmProjectOperator.releaseRoleBindings.clusterRoleRefs.edit`
- `helmProjectOperator.releaseRoleBindings.clusterRoleRefs.view`
By default, these roleRefs will correspond to `admin`, `edit`, and `view` respectively, which are the [default Kubernetes user-facing roles](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#user-facing-roles).
:::note
For Rancher RBAC users, these [default Kubernetes user-facing roles](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#user-facing-roles) directly correlate to the `Project Owner`, `Project Member`, and `Read-Only` default Project Role Templates.
:::
If the `roleRef` matches, the Helm Project Operator will filter the `subjects` of the binding for all Users and Groups and use that to automatically construct a RoleBinding for each Role in the Project Release Namespace with the same name as the role and the following labels:
- `helm.cattle.io/project-helm-chart-role: {{ .Release.Name }}`
- `helm.cattle.io/project-helm-chart-role-aggregate-from: <admin|edit|view>`
By default, `rancher-project-monitoring`, the underlying chart deployed by Prometheus Federator, creates three default Roles per Project Release Namespace that provide `admin`, `edit`, and `view` users to permissions to view the Prometheus, Alertmanager, and Grafana UIs of the Project Monitoring Stack to provide least privilege. However, if a Cluster Admin would like to assign additional permissions to certain users, they can either directly assign RoleBindings in the Project Release Namespace to certain users or create Roles with the above two labels on them to allow Project Owners to control assigning those RBAC roles to users in their Project Registration namespaces.
docs/reference-guides/rancher-cluster-tools.md
+59
View File
@@ -0,0 +1,59 @@
---
title: Tools for Logging, Monitoring, and Visibility
weight: 2033
---
Rancher contains a variety of tools that aren't included in Kubernetes to assist in your DevOps operations. Rancher can integrate with external services to help your clusters run more efficiently. Tools are divided into following categories:
<!-- TOC -->
- [Logging](#logging)
- [Monitoring and Alerts](#monitoring-and-alerts)
- [Istio](#istio)
- [OPA Gatekeeper](#opa-gatekeeper)
- [CIS Scans](#cis-scans)
<!-- /TOC -->
# Logging
Logging is helpful because it allows you to:
- Capture and analyze the state of your cluster
- Look for trends in your environment
- Save your logs to a safe location outside of your cluster
- Stay informed of events like a container crashing, a pod eviction, or a node dying
- More easily debugg and troubleshoot problems
Rancher can integrate with Elasticsearch, splunk, kafka, syslog, and fluentd.
For more information, refer to the logging documentation [here.](../pages-for-subheaders/logging.md)
# Monitoring and Alerts
Using Rancher, you can monitor the state and processes of your cluster nodes, Kubernetes components, and software deployments through integration with [Prometheus](https://prometheus.io/), a leading open-source monitoring solution.
After monitoring is enabled, you can set up alerts and notifiers that provide the mechanism to receive them.
Notifiers are services that inform you of alert events. You can configure notifiers to send alert notifications to staff best suited to take corrective action. Notifications can be sent with Slack, email, PagerDuty, WeChat, and webhooks.
Alerts are rules that trigger those notifications. Before you can receive alerts, you must configure one or more notifier in Rancher. The scope for alerts can be set at either the cluster or project level.
For more information, refer to the monitoring documentation [here.](../pages-for-subheaders/monitoring-and-alerting.md)
# Istio
[Istio](https://istio.io/) is an open-source tool that makes it easier for DevOps teams to observe, control, troubleshoot, and secure the traffic within a complex network of microservices.
Rancher's integration with Istio was improved in Rancher v2.5.
For more information, refer to the Istio documentation [here.](../pages-for-subheaders/istio.md)
# OPA Gatekeeper
[OPA Gatekeeper](https://github.com/open-policy-agent/gatekeeper) is an open-source project that provides integration between OPA and Kubernetes to provide policy control via admission controller webhooks. For details on how to enable Gatekeeper in Rancher, refer to the [OPA Gatekeeper section.](../explanations/integrations-in-rancher/opa-gatekeeper.md)
# CIS Scans
Rancher can run a security scan to check whether Kubernetes is deployed according to security best practices as defined in the CIS Kubernetes Benchmark.
For more information, refer to the CIS scan documentation [here.](../pages-for-subheaders/cis-scan-guides.md)
docs/reference-guides/rancher-manager-architecture/architecture-recommendations.md
+113
View File
@@ -0,0 +1,113 @@
---
title: Architecture Recommendations
weight: 3
---
If you are installing Rancher on a single node, the main architecture recommendation that applies to your installation is that the node running Rancher should be [separate from downstream clusters.](#separation-of-rancher-and-user-clusters)
This section covers the following topics:
- [Separation of Rancher and User Clusters](#separation-of-rancher-and-user-clusters)
- [Why HA is Better for Rancher in Production](#why-ha-is-better-for-rancher-in-production)
- [Recommended Load Balancer Configuration for Kubernetes Installations](#recommended-load-balancer-configuration-for-kubernetes-installations)
- [Environment for Kubernetes Installations](#environment-for-kubernetes-installations)
- [Recommended Node Roles for Kubernetes Installations](#recommended-node-roles-for-kubernetes-installations)
- [Architecture for an Authorized Cluster Endpoint (ACE)](#architecture-for-an-authorized-cluster-endpoint-ace)
# Separation of Rancher and User Clusters
A user cluster is a downstream Kubernetes cluster that runs your apps and services.
If you have a Docker installation of Rancher, the node running the Rancher server should be separate from your downstream clusters.
If Rancher is intended to manage downstream Kubernetes clusters, the Kubernetes cluster that the Rancher server runs on should also be separate from the downstream user clusters.
![Separation of Rancher Server from User Clusters](/img/rancher-architecture-separation-of-rancher-server.svg)
# Why HA is Better for Rancher in Production
We recommend installing the Rancher server on a high-availability Kubernetes cluster, primarily because it protects the Rancher server data. In a high-availability installation, a load balancer serves as the single point of contact for clients, distributing network traffic across multiple servers in the cluster and helping to prevent any one server from becoming a point of failure.
We don't recommend installing Rancher in a single Docker container, because if the node goes down, there is no copy of the cluster data available on other nodes and you could lose the data on your Rancher server.
### K3s Kubernetes Cluster Installations
One option for the underlying Kubernetes cluster is to use K3s Kubernetes. K3s is Rancher's CNCF certified Kubernetes distribution. It is easy to install and uses half the memory of Kubernetes, all in a binary of less than 100 MB. Another advantage of K3s is that it allows an external datastore to hold the cluster data, allowing the K3s server nodes to be treated as ephemeral.
<figcaption>Architecture of a K3s Kubernetes Cluster Running the Rancher Management Server</figcaption>
![Architecture of a K3s Kubernetes Cluster Running the Rancher Management Server](/img/k3s-server-storage.svg)
### RKE Kubernetes Cluster Installations
In an RKE installation, the cluster data is replicated on each of three etcd nodes in the cluster, providing redundancy and data duplication in case one of the nodes fails.
<figcaption>Architecture of an RKE Kubernetes Cluster Running the Rancher Management Server</figcaption>
![Architecture of an RKE Kubernetes cluster running the Rancher management server](/img/rke-server-storage.svg)
# Recommended Load Balancer Configuration for Kubernetes Installations
We recommend the following configurations for the load balancer and Ingress controllers:
* The DNS for Rancher should resolve to a Layer 4 load balancer (TCP).
* The Load Balancer should forward port TCP/80 and TCP/443 to all 3 nodes in the Kubernetes cluster.
* The Ingress controller will redirect HTTP to HTTPS and terminate SSL/TLS on port TCP/443.
* The Ingress controller will forward traffic to port TCP/80 on the pod in the Rancher deployment.
<figcaption>Rancher installed on a Kubernetes cluster with layer 4 load balancer, depicting SSL termination at Ingress controllers</figcaption>
![Rancher HA](/img/ha/rancher2ha.svg)
# Environment for Kubernetes Installations
It is strongly recommended to install Rancher on a Kubernetes cluster on hosted infrastructure such as Amazon's EC2 or Google Compute Engine.
For the best performance and greater security, we recommend a dedicated Kubernetes cluster for the Rancher management server. Running user workloads on this cluster is not advised. After deploying Rancher, you can [create or import clusters](../../pages-for-subheaders/kubernetes-clusters-in-rancher-setup.md) for running your workloads.
# Recommended Node Roles for Kubernetes Installations
The below recommendations apply when Rancher is installed on a K3s Kubernetes cluster or an RKE Kubernetes cluster.
### K3s Cluster Roles
In K3s clusters, there are two types of nodes: server nodes and agent nodes. Both servers and agents can have workloads scheduled on them. Server nodes run the Kubernetes master.
For the cluster running the Rancher management server, we recommend using two server nodes. Agent nodes are not required.
### RKE Cluster Roles
If Rancher is installed on an RKE Kubernetes cluster, the cluster should have three nodes, and each node should have all three Kubernetes roles: etcd, controlplane, and worker.
### Contrasting RKE Cluster Architecture for Rancher Server and for Downstream Kubernetes Clusters
Our recommendation for RKE node roles on the Rancher server cluster contrasts with our recommendations for the downstream user clusters that run your apps and services.
Rancher uses RKE as a library when provisioning downstream Kubernetes clusters. Note: The capability to provision downstream K3s clusters will be added in a future version of Rancher.
For downstream Kubernetes clusters, we recommend that each node in a user cluster should have a single role for stability and scalability.
![Kubernetes Roles for Nodes in Rancher Server Cluster vs. User Clusters](/img/rancher-architecture-node-roles.svg)
RKE only requires at least one node with each role and does not require nodes to be restricted to one role. However, for the clusters that run your apps, we recommend separate roles for each node so that workloads on worker nodes don't interfere with the Kubernetes master or cluster data as your services scale.
We recommend that downstream user clusters should have at least:
- **Three nodes with only the etcd role** to maintain a quorum if one node is lost, making the state of your cluster highly available
- **Two nodes with only the controlplane role** to make the master component highly available
- **One or more nodes with only the worker role** to run the Kubernetes node components, as well as the workloads for your apps and services
With that said, it is safe to use all three roles on three nodes when setting up the Rancher server because:
* It allows one `etcd` node failure.
* It maintains multiple instances of the master components by having multiple `controlplane` nodes.
* No other workloads than Rancher itself should be created on this cluster.
Because no additional workloads will be deployed on the Rancher server cluster, in most cases it is not necessary to use the same architecture that we recommend for the scalability and reliability of downstream clusters.
For more best practices for downstream clusters, refer to the [production checklist](../../pages-for-subheaders/checklist-for-production-ready-clusters.md) or our [best practices guide.](../../pages-for-subheaders/best-practices.md)
# Architecture for an Authorized Cluster Endpoint (ACE)
If you are using an [authorized cluster endpoint (ACE),](../../pages-for-subheaders/rancher-manager-architecture.md#4-authorized-cluster-endpoint) we recommend creating an FQDN pointing to a load balancer which balances traffic across your nodes with the `controlplane` role.
If you are using private CA signed certificates on the load balancer, you have to supply the CA certificate, which will be included in the generated kubeconfig file to validate the certificate chain. See the documentation on [kubeconfig files](../../how-to-guides/advanced-user-guides/manage-clusters/access-clusters/use-kubectl-and-kubeconfig.md) and [API keys](../user-settings/api-keys.md#creating-an-api-key) for more information.
As of Rancher v2.6.3, ACE support is available for registered RKE2 and K3s clusters. To view the manual steps to perform on the downstream cluster to enable the ACE, click [here](../../how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/register-existing-clusters.md#authorized-cluster-endpoint-support-for-rke2-and-k3s-clusters).
docs/reference-guides/rancher-manager-architecture/communicating-with-downstream-user-clusters.md
+1
View File
@@ -0,0 +1 @@
<!-- PLACEHOLDER -->
docs/reference-guides/rancher-manager-architecture/rancher-server-and-components.md
+1
View File
@@ -0,0 +1 @@
<!-- PLACEHOLDER -->
docs/reference-guides/rancher-project-tools.md
+39
View File
@@ -0,0 +1,39 @@
---
title: Tools for Logging, Monitoring, and Visibility
weight: 2525
---
Rancher contains a variety of tools that aren't included in Kubernetes to assist in your DevOps operations. Rancher can integrate with external services to help your clusters run more efficiently. Tools are divided into following categories:
<!-- TOC -->
- [Notifiers and Alerts](#notifiers-and-alerts)
- [Logging](#logging)
- [Monitoring](#monitoring)
<!-- /TOC -->
## Notifiers and Alerts
Notifiers and alerts are two features that work together to inform you of events in the Rancher system. Before they can be enabled, the monitoring application must be installed.
Notifiers are services that inform you of alert events. You can configure notifiers to send alert notifications to staff best suited to take corrective action. Notifications can be sent with Slack, email, PagerDuty, WeChat, and webhooks.
Alerts are rules that trigger those notifications. Before you can receive alerts, you must configure one or more notifier in Rancher. The scope for alerts can be set at either the cluster or project level.
## Logging
Logging is helpful because it allows you to:
- Capture and analyze the state of your cluster
- Look for trends in your environment
- Save your logs to a safe location outside of your cluster
- Stay informed of events like a container crashing, a pod eviction, or a node dying
- More easily debugg and troubleshoot problems
Rancher can integrate with Elasticsearch, splunk, kafka, syslog, and fluentd.
For details, refer to the [logging section.](../pages-for-subheaders/logging.md)
## Monitoring
Using Rancher, you can monitor the state and processes of your cluster nodes, Kubernetes components, and software deployments through integration with [Prometheus](https://prometheus.io/), a leading open-source monitoring solution. For details, refer to the [monitoring section.](../pages-for-subheaders/monitoring-and-alerting.md)
docs/reference-guides/rancher-security/kubernetes-security-best-practices.md
+12
View File
@@ -0,0 +1,12 @@
---
title: Kubernetes Security Best Practices
weight: 5
---
### Restricting cloud metadata API access
Cloud providers such as AWS, Azure, DigitalOcean or GCP often expose metadata services locally to instances. By default, this endpoint is accessible by pods running on a cloud instance, including pods in hosted Kubernetes providers such as EKS, AKS, DigitalOcean Kubernetes or GKE, and can contain cloud credentials for that node, provisioning data such as kubelet credentials, or other sensitive data. To mitigate this risk when running on a cloud platform, follow the [Kubernetes security recommendations](https://kubernetes.io/docs/tasks/administer-cluster/securing-a-cluster/#restricting-cloud-metadata-api-access): limit permissions given to instance credentials, use network policies to restrict pod access to the metadata API, and avoid using provisioning data to deliver secrets.
It is advised to consult your cloud provider's security best practices for further recommendations and specific details on how to restrict access to cloud instance metadata API.
Further references: MITRE ATT&CK knowledge base on - [Unsecured Credentials: Cloud Instance Metadata API](https://attack.mitre.org/techniques/T1552/005/).
docs/reference-guides/rancher-security/rancher-v2.6-hardening-guides/rke1-hardening-guide-with-cis-v1.6-benchmark.md
+655
View File
@@ -0,0 +1,655 @@
---
title: RKE Hardening Guide with CIS v1.6 Benchmark
weight: 100
aliases:
- /rancher/v2.6/en/security/hardening-guides/1.6-hardening-2.6/
---
This document provides prescriptive guidance for hardening a production installation of a RKE cluster to be used with Rancher v2.6. It outlines the configurations and controls required to address Kubernetes benchmark controls from the Center for Information Security (CIS).
:::note
This hardening guide describes how to secure the nodes in your cluster, and it is recommended to follow this guide before installing Kubernetes.
:::
This hardening guide is intended to be used for RKE clusters and associated with specific versions of the CIS Kubernetes Benchmark, Kubernetes, and Rancher:
| Rancher Version | CIS Benchmark Version | Kubernetes Version |
| --------------- | --------------------- | ------------------ |
| Rancher v2.6 | Benchmark v1.6 | Kubernetes v1.18 up to v1.23 |
[Click here to download a PDF version of this document](https://releases.rancher.com/documents/security/2.6/Rancher_v2-6_CIS_v1-6_Hardening_Guide.pdf).
- [Overview](#overview)
- [Configure Kernel Runtime Parameters](#configure-kernel-runtime-parameters)
- [Configure `etcd` user and group](#configure-etcd-user-and-group)
- [Configure `default` service account](#configure-default-service-account)
- [Configure Network Policy](#configure-network-policy)
- [Reference Hardened RKE `cluster.yml` Configuration](#reference-hardened-rke-cluster-yml-configuration)
- [Reference Hardened RKE Template Configuration](#reference-hardened-rke-template-configuration)
- [Reference Hardened **cloud-config** Configuration](#reference-hardened-cloud-config-configuration)
### Overview
This document provides prescriptive guidance for hardening a RKE cluster to be used for installing Rancher v2.6 with Kubernetes v1.18 up to v1.23 or provisioning a RKE cluster with Kubernetes v1.18 up to v1.23 to be used within Rancher v2.6. It outlines the configurations required to address Kubernetes benchmark controls from the Center for Information Security (CIS).
For more details about evaluating a hardened cluster against the official CIS benchmark, refer to the [CIS 1.6 Benchmark - Self-Assessment Guide - Rancher v2.6](security/hardening-guides/1.6-benchmark-2.6/).
#### Known Issues
- Rancher **exec shell** and **view logs** for pods are **not** functional in a CIS v1.6 hardened setup when only public IP is provided when registering custom nodes. This functionality requires a private IP to be provided when registering the custom nodes.
- When setting the `default_pod_security_policy_template_id:` to `restricted` or `restricted-noroot`, based on the pod security policies (PSP) [provided](../../../how-to-guides/advanced-user-guides/authentication-permissions-and-global-configuration/create-pod-security-policies.md) by Rancher, Rancher creates **RoleBindings** and **ClusterRoleBindings** on the default service accounts. The CIS v1.6 check 5.1.5 requires that the default service accounts have no roles or cluster roles bound to it apart from the defaults. In addition the default service accounts should be configured such that it does not provide a service account token and does not have any explicit rights assignments.
### Configure Kernel Runtime Parameters
The following `sysctl` configuration is recommended for all nodes type in the cluster. Set the following parameters in `/etc/sysctl.d/90-kubelet.conf`:
```ini
vm.overcommit_memory=1
vm.panic_on_oom=0
kernel.panic=10
kernel.panic_on_oops=1
kernel.keys.root_maxbytes=25000000
```
Run `sysctl -p /etc/sysctl.d/90-kubelet.conf` to enable the settings.
### Configure `etcd` user and group
A user account and group for the **etcd** service is required to be setup before installing RKE. The **uid** and **gid** for the **etcd** user will be used in the RKE **config.yml** to set the proper permissions for files and directories during installation time.
#### Create `etcd` user and group
To create the **etcd** user and group run the following console commands. The commands below use `52034` for **uid** and **gid** are for example purposes. Any valid unused **uid** or **gid** could also be used in lieu of `52034`.
```bash
groupadd --gid 52034 etcd
useradd --comment "etcd service account" --uid 52034 --gid 52034 etcd --shell /usr/sbin/nologin
```
Update the RKE **config.yml** with the **uid** and **gid** of the **etcd** user:
```yaml
services:
etcd:
gid: 52034
uid: 52034
```
### Configure `default` Service Account
#### Set `automountServiceAccountToken` to `false` for `default` service accounts
Kubernetes provides a default service account which is used by cluster workloads where no specific service account is assigned to the pod. Where access to the Kubernetes API from a pod is required, a specific service account should be created for that pod, and rights granted to that service account. The default service account should be configured such that it does not provide a service account token and does not have any explicit rights assignments.
For each namespace including **default** and **kube-system** on a standard RKE install, the **default** service account must include this value:
```yaml
automountServiceAccountToken: false
```
Save the following configuration to a file called `account_update.yaml`.
```yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: default
automountServiceAccountToken: false
```
Create a bash script file called `account_update.sh`. Be sure to `chmod +x account_update.sh` so the script has execute permissions.
```bash
#!/bin/bash -e
for namespace in $(kubectl get namespaces -A -o=jsonpath="{.items[*]['metadata.name']}"); do
kubectl patch serviceaccount default -n ${namespace} -p "$(cat account_update.yaml)"
done
```
### Configure Network Policy
#### Ensure that all Namespaces have Network Policies defined
Running different applications on the same Kubernetes cluster creates a risk of one compromised application attacking a neighboring application. Network segmentation is important to ensure that containers can communicate only with those they are supposed to. A network policy is a specification of how selections of pods are allowed to communicate with each other and other network endpoints.
Network Policies are namespace scoped. When a network policy is introduced to a given namespace, all traffic not allowed by the policy is denied. However, if there are no network policies in a namespace all traffic will be allowed into and out of the pods in that namespace. To enforce network policies, a CNI (container network interface) plugin must be enabled. This guide uses [Canal](https://github.com/projectcalico/canal) to provide the policy enforcement. Additional information about CNI providers can be found [here](https://www.suse.com/c/rancher_blog/comparing-kubernetes-cni-providers-flannel-calico-canal-and-weave/).
Once a CNI provider is enabled on a cluster a default network policy can be applied. For reference purposes a **permissive** example is provided below. If you want to allow all traffic to all pods in a namespace (even if policies are added that cause some pods to be treated as “isolated”), you can create a policy that explicitly allows all traffic in that namespace. Save the following configuration as `default-allow-all.yaml`. Additional [documentation](https://kubernetes.io/docs/concepts/services-networking/network-policies/) about network policies can be found on the Kubernetes site.
:::note
This `NetworkPolicy` is just an example and is not recommended for production use.
:::
```yaml
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-allow-all
spec:
podSelector: {}
ingress:
- {}
egress:
- {}
policyTypes:
- Ingress
- Egress
```
Create a bash script file called `apply_networkPolicy_to_all_ns.sh`. Be sure to `chmod +x apply_networkPolicy_to_all_ns.sh` so the script has execute permissions.
```bash
#!/bin/bash -e
for namespace in $(kubectl get namespaces -A -o=jsonpath="{.items[*]['metadata.name']}"); do
kubectl apply -f default-allow-all.yaml -n ${namespace}
done
```
Execute this script to apply the `default-allow-all.yaml` configuration with the **permissive** `NetworkPolicy` to all namespaces.
### Reference Hardened RKE `cluster.yml` Configuration
The reference `cluster.yml` is used by the RKE CLI that provides the configuration needed to achieve a hardened install of Rancher Kubernetes Engine (RKE). RKE install [documentation](https://rancher.com/docs/rke/latest/en/installation/) is provided with additional details about the configuration items. This reference `cluster.yml` does not include the required **nodes** directive which will vary depending on your environment. Documentation for node configuration in RKE can be found [here](https://rancher.com/docs/rke/latest/en/config-options/nodes/).
:::note Important:
For a Kubernetes v1.18 cluster, the configuration `spec.volumes: 'ephemeral'` should be removed from the `PodSecurityPolicy`, since it's not supported in this Kubernetes release.
:::
```yaml
# If you intend to deploy Kubernetes in an air-gapped environment,
# please consult the documentation on how to configure custom RKE images.
# https://rancher.com/docs/rke/latest/en/installation/ .
# The nodes directive is required and will vary depending on your environment.
# Documentation for node configuration can be found here:
# https://rancher.com/docs/rke/latest/en/config-options/nodes/
nodes: []
services:
etcd:
image: ""
extra_args: {}
extra_binds: []
extra_env: []
win_extra_args: {}
win_extra_binds: []
win_extra_env: []
external_urls: []
ca_cert: ""
cert: ""
key: ""
path: ""
uid: 52034
gid: 52034
snapshot: false
retention: ""
creation: ""
backup_config: null
kube-api:
image: ""
extra_args: {}
extra_binds: []
extra_env: []
win_extra_args: {}
win_extra_binds: []
win_extra_env: []
service_cluster_ip_range: ""
service_node_port_range: ""
pod_security_policy: true
always_pull_images: false
secrets_encryption_config:
enabled: true
custom_config: null
audit_log:
enabled: true
configuration: null
admission_configuration: null
event_rate_limit:
enabled: true
configuration: null
kube-controller:
image: ""
extra_args:
feature-gates: RotateKubeletServerCertificate=true
tls-cipher-suites: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256
bind-address: 127.0.0.1
extra_binds: []
extra_env: []
win_extra_args: {}
win_extra_binds: []
win_extra_env: []
cluster_cidr: ""
service_cluster_ip_range: ""
scheduler:
image: ""
extra_args:
tls-cipher-suites: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256
bind-address: 127.0.0.1
extra_binds: []
extra_env: []
win_extra_args: {}
win_extra_binds: []
win_extra_env: []
kubelet:
image: ""
extra_args:
feature-gates: RotateKubeletServerCertificate=true
protect-kernel-defaults: true
tls-cipher-suites: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256
extra_binds: []
extra_env: []
win_extra_args: {}
win_extra_binds: []
win_extra_env: []
cluster_domain: cluster.local
infra_container_image: ""
cluster_dns_server: ""
fail_swap_on: false
generate_serving_certificate: true
kubeproxy:
image: ""
extra_args: {}
extra_binds: []
extra_env: []
win_extra_args: {}
win_extra_binds: []
win_extra_env: []
network:
plugin: ""
options: {}
mtu: 0
node_selector: {}
update_strategy: null
authentication:
strategy: ""
sans: []
webhook: null
addons: |
# Upstream Kubernetes restricted PSP policy
# https://github.com/kubernetes/website/blob/564baf15c102412522e9c8fc6ef2b5ff5b6e766c/content/en/examples/policy/restricted-psp.yaml
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
name: restricted-noroot
spec:
privileged: false
# Required to prevent escalations to root.
allowPrivilegeEscalation: false
requiredDropCapabilities:
- ALL
# Allow core volume types.
volumes:
- 'configMap'
- 'emptyDir'
- 'projected'
- 'secret'
- 'downwardAPI'
# Assume that ephemeral CSI drivers & persistentVolumes set up by the cluster admin are safe to use.
- 'csi'
- 'persistentVolumeClaim'
- 'ephemeral'
hostNetwork: false
hostIPC: false
hostPID: false
runAsUser:
# Require the container to run without root privileges.
rule: 'MustRunAsNonRoot'
seLinux:
# This policy assumes the nodes are using AppArmor rather than SELinux.
rule: 'RunAsAny'
supplementalGroups:
rule: 'MustRunAs'
ranges:
# Forbid adding the root group.
- min: 1
max: 65535
fsGroup:
rule: 'MustRunAs'
ranges:
# Forbid adding the root group.
- min: 1
max: 65535
readOnlyRootFilesystem: false
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: psp:restricted-noroot
rules:
- apiGroups:
- extensions
resourceNames:
- restricted-noroot
resources:
- podsecuritypolicies
verbs:
- use
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: psp:restricted-noroot
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: psp:restricted-noroot
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: Group
name: system:serviceaccounts
- apiGroup: rbac.authorization.k8s.io
kind: Group
name: system:authenticated
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-allow-all
spec:
podSelector: {}
ingress:
- {}
egress:
- {}
policyTypes:
- Ingress
- Egress
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: default
automountServiceAccountToken: false
addons_include: []
system_images:
etcd: ""
alpine: ""
nginx_proxy: ""
cert_downloader: ""
kubernetes_services_sidecar: ""
kubedns: ""
dnsmasq: ""
kubedns_sidecar: ""
kubedns_autoscaler: ""
coredns: ""
coredns_autoscaler: ""
nodelocal: ""
kubernetes: ""
flannel: ""
flannel_cni: ""
calico_node: ""
calico_cni: ""
calico_controllers: ""
calico_ctl: ""
calico_flexvol: ""
canal_node: ""
canal_cni: ""
canal_controllers: ""
canal_flannel: ""
canal_flexvol: ""
weave_node: ""
weave_cni: ""
pod_infra_container: ""
ingress: ""
ingress_backend: ""
metrics_server: ""
windows_pod_infra_container: ""
ssh_key_path: ""
ssh_cert_path: ""
ssh_agent_auth: false
authorization:
mode: ""
options: {}
ignore_docker_version: false
kubernetes_version: ""
private_registries: []
ingress:
provider: ""
options: {}
node_selector: {}
extra_args: {}
dns_policy: ""
extra_envs: []
extra_volumes: []
extra_volume_mounts: []
update_strategy: null
http_port: 0
https_port: 0
network_mode: ""
cluster_name:
cloud_provider:
name: ""
prefix_path: ""
win_prefix_path: ""
addon_job_timeout: 0
bastion_host:
address: ""
port: ""
user: ""
ssh_key: ""
ssh_key_path: ""
ssh_cert: ""
ssh_cert_path: ""
monitoring:
provider: ""
options: {}
node_selector: {}
update_strategy: null
replicas: null
restore:
restore: false
snapshot_name: ""
dns: null
upgrade_strategy:
max_unavailable_worker: ""
max_unavailable_controlplane: ""
drain: null
node_drain_input: null
```
### Reference Hardened RKE Template Configuration
The reference RKE template provides the configuration needed to achieve a hardened install of Kubernetes. RKE templates are used to provision Kubernetes and define Rancher settings. Follow the Rancher [documentation](../../../pages-for-subheaders/installation-and-upgrade.md) for additional installation and RKE template details.
```yaml
#
# Cluster Config
#
default_pod_security_policy_template_id: restricted-noroot
docker_root_dir: /var/lib/docker
enable_cluster_alerting: false
enable_cluster_monitoring: false
enable_network_policy: true
local_cluster_auth_endpoint:
enabled: true
name: ''
#
# Rancher Config
#
rancher_kubernetes_engine_config:
addon_job_timeout: 45
authentication:
strategy: x509
dns:
nodelocal:
ip_address: ''
node_selector: null
update_strategy: {}
enable_cri_dockerd: false
ignore_docker_version: true
#
# # Currently only nginx ingress provider is supported.
# # To disable ingress controller, set `provider: none`
# # To enable ingress on specific nodes, use the node_selector, eg:
# provider: nginx
# node_selector:
# app: ingress
#
ingress:
default_backend: false
default_ingress_class: true
http_port: 0
https_port: 0
provider: nginx
kubernetes_version: v1.21.8-rancher1-1
monitoring:
provider: metrics-server
replicas: 1
#
# If you are using calico on AWS
#
# network:
# plugin: calico
# calico_network_provider:
# cloud_provider: aws
#
# # To specify flannel interface
#
# network:
# plugin: flannel
# flannel_network_provider:
# iface: eth1
#
# # To specify flannel interface for canal plugin
#
# network:
# plugin: canal
# canal_network_provider:
# iface: eth1
#
network:
mtu: 0
options:
flannel_backend_type: vxlan
plugin: canal
rotate_encryption_key: false
#
# services:
# kube-api:
# service_cluster_ip_range: 10.43.0.0/16
# kube-controller:
# cluster_cidr: 10.42.0.0/16
# service_cluster_ip_range: 10.43.0.0/16
# kubelet:
# cluster_domain: cluster.local
# cluster_dns_server: 10.43.0.10
#
services:
scheduler:
extra_args:
tls-cipher-suites: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256
bind-address: 127.0.0.1
etcd:
backup_config:
enabled: true
interval_hours: 12
retention: 6
safe_timestamp: false
timeout: 300
creation: 12h
extra_args:
election-timeout: 5000
heartbeat-interval: 500
retention: 72h
snapshot: false
uid: 52034
gid: 52034
kube_api:
always_pull_images: false
audit_log:
enabled: true
event_rate_limit:
enabled: true
pod_security_policy: true
secrets_encryption_config:
enabled: true
service_node_port_range: 30000-32767
kube-controller:
extra_args:
feature-gates: RotateKubeletServerCertificate=true
tls-cipher-suites: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256
bind-address: 127.0.0.1
kubelet:
extra_args:
feature-gates: RotateKubeletServerCertificate=true
protect-kernel-defaults: true
tls-cipher-suites: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256
fail_swap_on: false
generate_serving_certificate: true
ssh_agent_auth: false
upgrade_strategy:
max_unavailable_controlplane: '1'
max_unavailable_worker: 10%
windows_prefered_cluster: false
```
### Reference Hardened **cloud-config** Configuration
A **cloud-config** configuration file is generally used in cloud infrastructure environments to allow for configuration management of compute instances. The reference config configures SUSE Linux Enterprise Server (SLES), openSUSE Leap, Red Hat Enterprise Linux (RHEL) and Ubuntu operating system level settings needed before installing Kubernetes.
#### Reference Hardened **cloud-config** for SUSE Linux Enterprise Server 15 (SLES 15) and openSUSE Leap 15
```yaml
#cloud-config
system_info:
default_user:
groups:
- docker
write_files:
- path: "/etc/sysctl.d/90-kubelet.conf"
owner: root:root
permissions: '0644'
content: |
vm.overcommit_memory=1
vm.panic_on_oom=0
kernel.panic=10
kernel.panic_on_oops=1
kernel.keys.root_maxbytes=25000000
package_update: true
ssh_pwauth: false
runcmd:
# Docker should already be installed in SLES 15 SP3
- zypper install docker containerd
- systemctl daemon-reload
- systemctl enable docker.service
- systemctl start --no-block docker.service
- sysctl -p /etc/sysctl.d/90-kubelet.conf
- groupadd --gid 52034 etcd
- useradd --comment "etcd service account" --uid 52034 --gid 52034 etcd --shell /usr/sbin/nologin
```
#### Reference Hardened **cloud-config** for Red Hat Enterprise Linux 8 (RHEL 8) and Ubuntu 20.04 LTS
```yaml
#cloud-config
system_info:
default_user:
groups:
- docker
write_files:
- path: "/etc/sysctl.d/90-kubelet.conf"
owner: root:root
permissions: '0644'
content: |
vm.overcommit_memory=1
vm.panic_on_oom=0
kernel.panic=10
kernel.panic_on_oops=1
kernel.keys.root_maxbytes=25000000
package_update: true
ssh_pwauth: false
runcmd:
# Install Docker from Rancher's Docker installation scripts - github.com/rancher/install-docker
- curl https://releases.rancher.com/install-docker/20.10.sh | sh
- sysctl -p /etc/sysctl.d/90-kubelet.conf
- groupadd --gid 52034 etcd
- useradd --comment "etcd service account" --uid 52034 --gid 52034 etcd --shell /usr/sbin/nologin
```
docs/reference-guides/rancher-security/rancher-v2.6-hardening-guides/rke1-self-assessment-guide-with-cis-v1.6-benchmark.md
+3104
View File
File diff suppressed because it is too large Load Diff
docs/reference-guides/rancher-security/rancher-v2.6-hardening-guides/rke2-hardening-guide-with-cis-v1.6-benchmark.md
+418
View File
@@ -0,0 +1,418 @@
---
title: RKE2 Hardening Guide with CIS v1.6 Benchmark
weight: 100
---
This document provides prescriptive guidance for hardening a production installation of a RKE2 cluster to be provisioned with Rancher v2.6.5. It outlines the configurations and controls required to address Kubernetes benchmark controls from the Center for Information Security (CIS).
:::note
This hardening guide describes how to secure the nodes in your cluster, and it is recommended to follow this guide before installing Kubernetes.
:::
This hardening guide is intended to be used for RKE2 clusters and associated with specific versions of the CIS Kubernetes Benchmark, Kubernetes, and Rancher:
| Rancher Version | CIS Benchmark Version | Kubernetes Version |
| --------------- | --------------------- | ------------------ |
| Rancher v2.6.5+ | Benchmark v1.6 | Kubernetes v1.21 up to v1.23 |
[Click here to download a PDF version of this document](https://releases.rancher.com/documents/security/2.6/Rancher_RKE2_v2-6_CIS_v1-6_Hardening_Guide.pdf).
- [Overview](#overview)
- [Host-level requirements](#host-level-requirements)
- [Setting up hosts](#setting-up-hosts)
- [Kubernetes runtime requirements](#kubernetes-runtime-requirements)
- [API Server audit configuration](#api-server-audit-configuration)
- [Known issues](#known-issues)
- [Reference Hardened RKE2 Template Configuration](#reference-hardened-rke2-template-configuration)
- [Conclusion](#conclusion)
### Overview
This document provides prescriptive guidance for hardening a RKE2 cluster to be provisioned through Rancher v2.6.5+ with Kubernetes v1.21 up to v1.23. It outlines the configurations required to address Kubernetes benchmark controls from the Center for Information Security (CIS).
For more details about evaluating a hardened RKE2 cluster against the official CIS benchmark, refer to the [RKE2 - CIS 1.6 Benchmark - Self-Assessment Guide - Rancher v2.6](rke2-self-assessment-guide-with-cis-v1.6-benchmark.md).
RKE2 is designed to be "hardened by default" and pass the majority of the Kubernetes CIS controls without modification. There are a few notable exceptions to this that require manual intervention to fully pass the CIS Benchmark:
1. RKE2 will not modify the host operating system. Therefore, you, the operator, must make a few host-level modifications.
2. Certain CIS policy controls for `PodSecurityPolicies` and `NetworkPolicies` will restrict the functionality of the cluster. You must opt into having RKE2 configuring these out of the box.
To help ensure these above requirements are met, RKE2 can be started with the `profile` flag set to `cis-1.6`. This flag generally does two things:
1. Checks that host-level requirements have been met. If they haven't, RKE2 will exit with a fatal error describing the unmet requirements.
2. Configures runtime pod security policies and network policies that allow the cluster to pass associated controls.
:::note
The profile's flag only valid values are `cis-1.5` or `cis-1.6`. It accepts a string value to allow for other profiles in the future.
:::
The following section outlines the specific actions that are taken when the `profile` flag is set to `cis-1.6`.
### Host-level requirements
There are two areas of host-level requirements: kernel parameters and etcd process/directory configuration. These are outlined in this section.
#### Ensure `protect-kernel-defaults` is set
This is a kubelet flag that will cause the kubelet to exit if the required kernel parameters are unset or are set to values that are different from the kubelet's defaults.
When the `profile` flag is set, RKE2 will set the flag to `true`.
:::caution
`protect-kernel-defaults` is exposed as a configuration flag for RKE2. If you have set `profile` to "cis-1.x" and `protect-kernel-defaults` to `false` explicitly, RKE2 will exit with an error.
:::
RKE2 will also check the same kernel parameters that the kubelet does and exit with an error following the same rules as the kubelet. This is done as a convenience to help the operator more quickly and easily identify what kernel parameters are violating the kubelet defaults.
Both `protect-kernel-defaults` and `profile` flags can be set in RKE2 template configuration file.
```yaml
spec:
rkeConfig:
machineSelectorConfig:
- config:
profile: cis-1.6
protect-kernel-defaults: true
```
#### Ensure etcd is configured properly
The CIS Benchmark requires that the etcd data directory be owned by the `etcd` user and group. This implicitly requires the etcd process to be ran as the host-level `etcd` user. To achieve this, RKE2 takes several steps when started with a valid "cis-1.x" profile:
1. Check that the `etcd` user and group exists on the host. If they don't, exit with an error.
2. Create etcd's data directory with `etcd` as the user and group owner.
3. Ensure the etcd process is ran as the `etcd` user and group by setting the etcd static pod's `SecurityContext` appropriately.
### Setting up hosts
This section gives you the commands necessary to configure your host to meet the above requirements.
#### Set kernel parameters
The following `sysctl` configuration is recommended for all nodes type in the cluster. Set the following parameters in `/etc/sysctl.d/90-kubelet.conf`:
```ini
vm.panic_on_oom=0
vm.overcommit_memory=1
kernel.panic=10
kernel.panic_on_oops=1
```
Run `sudo sysctl -p /etc/sysctl.d/90-kubelet.conf` to enable the settings.
Please perform this step only on fresh installations, before actually deploying RKE2 through Rancher.
#### Create the etcd user
On some Linux distributions, the `useradd` command will not create a group. The `-U` flag is included below to account for that. This flag tells `useradd` to create a group with the same name as the user.
```bash
sudo useradd -r -c "etcd user" -s /sbin/nologin -M etcd -U
```
### Kubernetes runtime requirements
The runtime requirements to pass the CIS Benchmark are centered around pod security and network policies. These are outlined in this section.
#### `PodSecurityPolicies`
RKE2 always runs with the `PodSecurityPolicy` admission controller turned on. However, when it is **not** started with a valid "cis-1.x" profile, RKE2 will put an unrestricted policy in place that allows Kubernetes to run as though the `PodSecurityPolicy` admission controller was not enabled.
When ran with a valid "cis-1.x" profile, RKE2 will put a much more restrictive set of policies in place. These policies meet the requirements outlined in section 5.2 of the CIS Benchmark.
> The Kubernetes control plane components and critical additions such as CNI, DNS, and Ingress are ran as pods in the `kube-system` namespace. Therefore, this namespace will have a policy that is less restrictive so that these components can run properly.
#### `NetworkPolicies`
When ran with a valid "cis-1.x" profile, RKE2 will put `NetworkPolicies` in place that passes the CIS Benchmark for Kubernetes' built-in namespaces. These namespaces are: `kube-system`, `kube-public`, `kube-node-lease`, and `default`.
The `NetworkPolicy` used will only allow pods within the same namespace to talk to each other. The notable exception to this is that it allows DNS requests to be resolved.
:::note
Operators must manage network policies as normal for additional namespaces that are created.
:::
#### Configure `default` service account
**Set `automountServiceAccountToken` to `false` for `default` service accounts**
Kubernetes provides a `default` service account which is used by cluster workloads where no specific service account is assigned to the pod. Where access to the Kubernetes API from a pod is required, a specific service account should be created for that pod, and rights granted to that service account. The `default` service account should be configured such that it does not provide a service account token and does not have any explicit rights assignments.
For each namespace including `default` and `kube-system` on a standard RKE2 install, the `default` service account must include this value:
```yaml
automountServiceAccountToken: false
```
For namespaces created by the cluster operator, the following script and configuration file can be used to configure the `default` service account.
The configuration bellow must be saved to a file called `account_update.yaml`.
```yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: default
automountServiceAccountToken: false
```
Create a bash script file called `account_update.sh`. Be sure to `sudo chmod +x account_update.sh` so the script has execute permissions.
```bash
#!/bin/bash -e
for namespace in $(kubectl get namespaces -A -o=jsonpath="{.items[*]['metadata.name']}"); do
echo -n "Patching namespace $namespace - "
kubectl patch serviceaccount default -n ${namespace} -p "$(cat account_update.yaml)"
done
```
Execute this script to apply the `account_update.yaml` configuration to `default` service account in all namespaces.
### API Server audit configuration
CIS requirements 1.2.22 to 1.2.25 are related to configuring audit logs for the API Server. When RKE2 is started with the `profile` flag set to `cis-1.6`, it will automatically configure hardened `--audit-log-` parameters in the API Server to pass those CIS checks.
RKE2's default audit policy is configured to not log requests in the API Server. This is done to allow cluster operators flexibility to customize an audit policy that suits their auditing requirements and needs, as these are specific to each users' environment and policies.
A default audit policy is created by RKE2 when started with the `profile` flag set to `cis-1.6`. The policy is defined in `/etc/rancher/rke2/audit-policy.yaml`.
```yaml
apiVersion: audit.k8s.io/v1
kind: Policy
metadata:
creationTimestamp: null
rules:
- level: None
```
To start logging requests to the API Server, at least `level` parameter must be modified, for example, to `Metadata`. Detailed information about policy configuration for the API server can be found in the Kubernetes [documentation](https://kubernetes.io/docs/tasks/debug-application-cluster/audit/).
After adapting the audit policy, RKE2 must be restarted to load the new configuration.
```shell
sudo systemctl restart rke2-server.service
```
API Server audit logs will be written to `/var/lib/rancher/rke2/server/logs/audit.log`.
### Known issues
The following are controls that RKE2 currently does not pass. Each gap will be explained and whether it can be passed through manual operator intervention or if it will be addressed in a future release.
#### Control 1.1.12
Ensure that the etcd data directory ownership is set to `etcd:etcd`.
**Rationale**
etcd is a highly-available key-value store used by Kubernetes deployments for persistent storage of all of its REST API objects. This data directory should be protected from any unauthorized reads or writes. It should be owned by `etcd:etcd`.
**Remediation**
This can be remediated by creating an `etcd` user and group as described above.
#### Control 5.1.5
Ensure that default service accounts are not actively used
**Rationale** Kubernetes provides a `default` service account which is used by cluster workloads where no specific service account is assigned to the pod.
Where access to the Kubernetes API from a pod is required, a specific service account should be created for that pod, and rights granted to that service account.
The `default` service account should be configured such that it does not provide a service account token and does not have any explicit rights assignments.
This can be remediated by updating the `automountServiceAccountToken` field to `false` for the `default` service account in each namespace.
**Remediation**
You can manually update this field on service accounts in your cluster to pass the control as described above.
#### Control 5.3.2
Ensure that all Namespaces have Network Policies defined
**Rationale**
Running different applications on the same Kubernetes cluster creates a risk of one compromised application attacking a neighboring application. Network segmentation is important to ensure that containers can communicate only with those they are supposed to. A network policy is a specification of how selections of pods are allowed to communicate with each other and other network endpoints.
Network Policies are namespace scoped. When a network policy is introduced to a given namespace, all traffic not allowed by the policy is denied. However, if there are no network policies in a namespace all traffic will be allowed into and out of the pods in that namespace.
**Remediation**
This can be remediated by setting `profile: "cis-1.6"` in RKE2 template configuration file. An example can be found below.
### Reference Hardened RKE2 Template Configuration
The reference template configuration is used in Rancher to create a hardened RKE2 custom cluster. This reference does not include other required **cluster configuration** directives which will vary depending on your environment.
```yaml
apiVersion: provisioning.cattle.io/v1
kind: Cluster
metadata:
name: <replace_with_cluster_name>
annotations:
{}
# key: string
labels:
{}
# key: string
namespace: fleet-default
spec:
defaultPodSecurityPolicyTemplateName: ''
kubernetesVersion: <replace_with_kubernetes_version>
localClusterAuthEndpoint:
caCerts: ''
enabled: false
fqdn: ''
rkeConfig:
chartValues:
rke2-canal:
{}
etcd:
disableSnapshots: false
s3:
# bucket: string
# cloudCredentialName: string
# endpoint: string
# endpointCA: string
# folder: string
# region: string
# skipSSLVerify: boolean
snapshotRetention: 5
snapshotScheduleCron: 0 */5 * * *
machineGlobalConfig:
cni: canal
machinePools:
# - cloudCredentialSecretName: string
# controlPlaneRole: boolean
# displayName: string
# drainBeforeDelete: boolean
# etcdRole: boolean
# labels:
# key: string
# machineConfigRef:
# apiVersion: string
# fieldPath: string
# kind: string
# name: string
# namespace: string
# resourceVersion: string
# uid: string
# machineDeploymentAnnotations:
# key: string
# machineDeploymentLabels:
# key: string
# machineOS: string
# maxUnhealthy: string
# name: string
# nodeStartupTimeout: string
# paused: boolean
# quantity: int
# rollingUpdate:
# maxSurge: string
# maxUnavailable: string
# taints:
# - effect: string
# key: string
# timeAdded: string
# value: string
# unhealthyNodeTimeout: string
# unhealthyRange: string
# workerRole: boolean
machineSelectorConfig:
- config:
profile: cis-1.6
protect-kernel-defaults: true
# - config:
#
# machineLabelSelector:
# matchExpressions:
# - key: string
# operator: string
# values:
# - string
# matchLabels:
# key: string
registries:
configs:
{}
#authConfigSecretName: string
# caBundle: string
# insecureSkipVerify: boolean
# tlsSecretName: string
mirrors:
{}
#endpoint:
# - string
# rewrite:
# key: string
upgradeStrategy:
controlPlaneConcurrency: 10%
controlPlaneDrainOptions:
# deleteEmptyDirData: boolean
# disableEviction: boolean
# enabled: boolean
# force: boolean
# gracePeriod: int
# ignoreDaemonSets: boolean
# ignoreErrors: boolean
# postDrainHooks:
# - annotation: string
# preDrainHooks:
# - annotation: string
# skipWaitForDeleteTimeoutSeconds: int
# timeout: int
workerConcurrency: 10%
workerDrainOptions:
# deleteEmptyDirData: boolean
# disableEviction: boolean
# enabled: boolean
# force: boolean
# gracePeriod: int
# ignoreDaemonSets: boolean
# ignoreErrors: boolean
# postDrainHooks:
# - annotation: string
# preDrainHooks:
# - annotation: string
# skipWaitForDeleteTimeoutSeconds: int
# timeout: int
# additionalManifest: string
# etcdSnapshotCreate:
# generation: int
# etcdSnapshotRestore:
# generation: int
# name: string
# restoreRKEConfig: string
# infrastructureRef:
# apiVersion: string
# fieldPath: string
# kind: string
# name: string
# namespace: string
# resourceVersion: string
# uid: string
# provisionGeneration: int
# rotateCertificates:
# generation: int
# services:
# - string
# rotateEncryptionKeys:
# generation: int
machineSelectorConfig:
- config: {}
# agentEnvVars:
# - name: string
# value: string
# cloudCredentialSecretName: string
# clusterAPIConfig:
# clusterName: string
# defaultClusterRoleForProjectMembers: string
# enableNetworkPolicy: boolean
# redeploySystemAgentGeneration: int
__clone: true
```
### Conclusion
If you have followed this guide, your RKE2 custom cluster provisioned by Rancher will be configured to pass the CIS Kubernetes Benchmark. You can review our RKE2 CIS Benchmark Self-Assessment Guide [v1.6](rke2-self-assessment-guide-with-cis-v1.6-benchmark.md) to understand how we verified each of the benchmarks and how you can do the same on your cluster.
docs/reference-guides/rancher-security/rancher-v2.6-hardening-guides/rke2-self-assessment-guide-with-cis-v1.6-benchmark.md
+3330
View File
File diff suppressed because it is too large Load Diff
docs/reference-guides/rancher-security/security-advisories-and-cves.md
+31
View File
@@ -0,0 +1,31 @@
---
title: Security Advisories and CVEs
weight: 300
aliases:
- /rancher/v2.x/en/security/cve/
---
Rancher is committed to informing the community of security issues in our products. Rancher will publish security advisories and CVEs (Common Vulnerabilities and Exposures) for issues we have resolved. New security advisories are also published in Rancher's GitHub [security page](https://github.com/rancher/rancher/security/advisories).
| ID | Description | Date | Resolution |
|----|-------------|------|------------|
| [CVE-2022-21951](https://github.com/rancher/rancher/security/advisories/GHSA-vrph-m5jj-c46c) | This vulnerability only affects customers using [Weave](https://rancher.com/docs/rancher/v2.6/en/faq/networking/cni-providers/#weave) Container Network Interface (CNI) when configured through [RKE templates](https://rancher.com/docs/rancher/v2.6/en/admin-settings/rke-templates/). A vulnerability was discovered in Rancher versions 2.5.0 up to and including 2.5.13, and 2.6.0 up to and including 2.6.4, where a user interface (UI) issue with RKE templates does not include a value for the Weave password when Weave is chosen as the CNI. If a cluster is created based on the mentioned template, and Weave is configured as the CNI, no password will be created for [network encryption](https://www.weave.works/docs/net/latest/tasks/manage/security-untrusted-networks/) in Weave; therefore, network traffic in the cluster will be sent unencrypted. | 24 May 2022 | [Rancher v2.6.5](https://github.com/rancher/rancher/releases/tag/v2.6.5) and [Rancher v2.5.14](https://github.com/rancher/rancher/releases/tag/v2.5.14) |
| [CVE-2021-36784](https://github.com/rancher/rancher/security/advisories/GHSA-jwvr-vv7p-gpwq) | A vulnerability was discovered in Rancher versions from 2.5.0 up to and including 2.5.12 and from 2.6.0 up to and including 2.6.3 which allows users who have create or update permissions on [Global Roles](https://rancher.com/docs/rancher/v2.6/en/admin-settings/rbac/) to escalate their permissions, or those of another user, to admin-level permissions. Global Roles grant users Rancher-wide permissions, such as the ability to create clusters. In the identified versions of Rancher, when users are given permission to edit or create Global Roles, they are not restricted to only granting permissions which they already posses. This vulnerability affects customers who utilize non-admin users that are able to create or edit Global Roles. The most common use case for this scenario is the `restricted-admin` role. | 14 Apr 2022 | [Rancher v2.6.4](https://github.com/rancher/rancher/releases/tag/v2.6.4) and [Rancher v2.5.13](https://github.com/rancher/rancher/releases/tag/v2.5.13) |
| [CVE-2021-4200](https://github.com/rancher/rancher/security/advisories/GHSA-hx8w-ghh8-r4xf) | This vulnerability only affects customers using the `restricted-admin` role in Rancher. A vulnerability was discovered in Rancher versions from 2.5.0 up to and including 2.5.12 and from 2.6.0 up to and including 2.6.3 where the `global-data` role in `cattle-global-data` namespace grants write access to the Catalogs. Since each user with any level of catalog access was bound to the `global-data` role, this grants write access to templates (`CatalogTemplates`) and template versions (`CatalogTemplateVersions`) for any user with any level of catalog access. New users created in Rancher are by default assigned to the `user` role (standard user), which is not designed to grant write catalog access. This vulnerability effectively elevates the privilege of any user to write access for the catalog template and catalog template version resources. | 14 Apr 2022 | [Rancher v2.6.4](https://github.com/rancher/rancher/releases/tag/v2.6.4) and [Rancher v2.5.13](https://github.com/rancher/rancher/releases/tag/v2.5.13) |
| [GHSA-wm2r-rp98-8pmh](https://github.com/rancher/rancher/security/advisories/GHSA-wm2r-rp98-8pmh) | This vulnerability only affects customers using [Fleet](https://rancher.com/docs/rancher/v2.6/en/deploy-across-clusters/fleet/) for continuous delivery with authenticated Git and/or Helm repositories. An issue was discovered in `go-getter` library in versions prior to [`v1.5.11`](https://github.com/hashicorp/go-getter/releases/tag/v1.5.11) that exposes SSH private keys in base64 format due to a failure in redacting such information from error messages. The vulnerable version of this library is used in Rancher through Fleet in versions of Fleet prior to [`v0.3.9`](https://github.com/rancher/fleet/releases/tag/v0.3.9). This issue affects Rancher versions 2.5.0 up to and including 2.5.12 and from 2.6.0 up to and including 2.6.3. The issue was found and reported by Dagan Henderson from Raft Engineering. | 14 Apr 2022 | [Rancher v2.6.4](https://github.com/rancher/rancher/releases/tag/v2.6.4) and [Rancher v2.5.13](https://github.com/rancher/rancher/releases/tag/v2.5.13) |
| [CVE-2021-36778](https://github.com/rancher/rancher/security/advisories/GHSA-4fc7-hc63-7fjg) | A vulnerability was discovered in Rancher versions from 2.5.0 up to and including 2.5.11 and from 2.6.0 up to and including 2.6.2, where an insufficient check of the same-origin policy when downloading Helm charts from a configured private repository can lead to exposure of the repository credentials to a third-party provider. This issue only happens when the user configures access credentials to a private repository in Rancher inside `Apps & Marketplace > Repositories`. The issue was found and reported by Martin Andreas Ullrich. | 14 Apr 2022 | [Rancher v2.6.3](https://github.com/rancher/rancher/releases/tag/v2.6.3) and [Rancher v2.5.12](https://github.com/rancher/rancher/releases/tag/v2.5.12) |
| [GHSA-hwm2-4ph6-w6m5](https://github.com/rancher/rancher/security/advisories/GHSA-hwm2-4ph6-w6m5) | A vulnerability was discovered in versions of Rancher starting 2.0 up to and including 2.6.3. The `restricted` pod security policy (PSP) provided in Rancher deviated from the upstream `restricted` policy provided in Kubernetes on account of which Rancher's PSP had `runAsUser` set to `runAsAny`, while upstream had `runAsUser` set to `MustRunAsNonRoot`. This allowed containers to run as any user, including a privileged user (`root`), even when Rancher's `restricted` policy was enforced on a project or at the cluster level. | 31 Mar 2022 | [Rancher v2.6.4](https://github.com/rancher/rancher/releases/tag/v2.6.4) |
| [CVE-2021-36775](https://github.com/rancher/rancher/security/advisories/GHSA-28g7-896h-695v) | A vulnerability was discovered in Rancher versions up to and including 2.4.17, 2.5.11 and 2.6.2. After removing a `Project Role` associated with a group from the project, the bindings that granted access to cluster-scoped resources for those subjects were not deleted. This was due to an incomplete authorization logic check. A user who was a member of the affected group with authenticated access to Rancher could exploit this vulnerability to access resources they shouldn't have had access to. The exposure level would depend on the original permission level granted to the affected project role. This vulnerability only affected customers using group based authentication in Rancher. | 31 Mar 2022 | [Rancher v2.6.3](https://github.com/rancher/rancher/releases/tag/v2.6.3), [Rancher v2.5.12](https://github.com/rancher/rancher/releases/tag/v2.5.12) and [Rancher v2.4.18](https://github.com/rancher/rancher/releases/tag/v2.4.18) |
| [CVE-2021-36776](https://github.com/rancher/rancher/security/advisories/GHSA-gvh9-xgrq-r8hw) | A vulnerability was discovered in Rancher versions starting 2.5.0 up to and including 2.5.9, that allowed an authenticated user to impersonate any user on a cluster through an API proxy, without requiring knowledge of the impersonated user's credentials. This was due to the API proxy not dropping the impersonation header before sending the request to the Kubernetes API. A malicious user with authenticated access to Rancher could use this to impersonate another user with administrator access in Rancher, thereby gaining administrator level access to the cluster. | 31 Mar 2022 | [Rancher v2.6.0](https://github.com/rancher/rancher/releases/tag/v2.6.0) and [Rancher v2.5.10](https://github.com/rancher/rancher/releases/tag/v2.5.10) |
| [CVE-2021-25318](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-25318) | A vulnerability was discovered in Rancher versions 2.0 through the aforementioned fixed versions, where users were granted access to resources regardless of the resource's API group. For example, Rancher should have allowed users access to `apps.catalog.cattle.io`, but instead incorrectly gave access to `apps.*`. Resources affected in the **Downstream clusters** and **Rancher management cluster** can be found [here](https://github.com/rancher/rancher/security/advisories/GHSA-f9xf-jq4j-vqw4). There is not a direct mitigation besides upgrading to the patched Rancher versions. | 14 Jul 2021 | [Rancher v2.5.9](https://github.com/rancher/rancher/releases/tag/v2.5.9) and [Rancher v2.4.16](https://github.com/rancher/rancher/releases/tag/v2.4.16) |
| [CVE-2021-31999](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-31999) | A vulnerability was discovered in Rancher 2.0.0 through the aforementioned patched versions, where a malicious Rancher user could craft an API request directed at the proxy for the Kubernetes API of a managed cluster to gain access to information they do not have access to. This is done by passing the "Impersonate-User" or "Impersonate-Group" header in the Connection header, which is then correctly removed by the proxy. At this point, instead of impersonating the user and their permissions, the request will act as if it was from the Rancher management server and incorrectly return the information. The vulnerability is limited to valid Rancher users with some level of permissions on the cluster. There is not a direct mitigation besides upgrading to the patched Rancher versions. | 14 Jul 2021 | [Rancher v2.5.9](https://github.com/rancher/rancher/releases/tag/v2.5.9) and [Rancher v2.4.16](https://github.com/rancher/rancher/releases/tag/v2.4.16) |
| [CVE-2021-25320](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-25320) | A vulnerability was discovered in Rancher 2.2.0 through the aforementioned patched versions, where cloud credentials weren't being properly validated through the Rancher API. Specifically through a proxy designed to communicate with cloud providers. Any Rancher user that was logged-in and aware of a cloud-credential ID that was valid for a given cloud provider, could call that cloud provider's API through the proxy API, and the cloud-credential would be attached. The exploit is limited to valid Rancher users. There is not a direct mitigation outside of upgrading to the patched Rancher versions. | 14 Jul 2021 | [Rancher v2.5.9](https://github.com/rancher/rancher/releases/tag/v2.5.9) and [Rancher v2.4.16](https://github.com/rancher/rancher/releases/tag/v2.4.16) |
| [CVE-2021-25313](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-25313) | A security vulnerability was discovered on all Rancher 2 versions. When accessing the Rancher API with a browser, the URL was not properly escaped, making it vulnerable to an XSS attack. Specially crafted URLs to these API endpoints could include JavaScript which would be embedded in the page and execute in a browser. There is no direct mitigation. Avoid clicking on untrusted links to your Rancher server. | 2 Mar 2021 | [Rancher v2.5.6](https://github.com/rancher/rancher/releases/tag/v2.5.6), [Rancher v2.4.14](https://github.com/rancher/rancher/releases/tag/v2.4.14), and [Rancher v2.3.11](https://github.com/rancher/rancher/releases/tag/v2.3.11) |
| [CVE-2019-14435](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-14435) | This vulnerability allows authenticated users to potentially extract otherwise private data out of IPs reachable from system service containers used by Rancher. This can include but not only limited to services such as cloud provider metadata services. Although Rancher allow users to configure whitelisted domains for system service access, this flaw can still be exploited by a carefully crafted HTTP request. The issue was found and reported by Matt Belisle and Alex Stevenson at Workiva. | 5 Aug 2019 | [Rancher v2.2.7](https://github.com/rancher/rancher/releases/tag/v2.2.7) and [Rancher v2.1.12](https://github.com/rancher/rancher/releases/tag/v2.1.12) |
| [CVE-2019-14436](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-14436) | The vulnerability allows a member of a project that has access to edit role bindings to be able to assign themselves or others a cluster level role granting them administrator access to that cluster. The issue was found and reported by Michal Lipinski at Nokia. | 5 Aug 2019 | [Rancher v2.2.7](https://github.com/rancher/rancher/releases/tag/v2.2.7) and [Rancher v2.1.12](https://github.com/rancher/rancher/releases/tag/v2.1.12) |
| [CVE-2019-13209](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-13209) | The vulnerability is known as a [Cross-Site Websocket Hijacking attack](https://www.christian-schneider.net/CrossSiteWebSocketHijacking.html). This attack allows an exploiter to gain access to clusters managed by Rancher with the roles/permissions of a victim. It requires that a victim to be logged into a Rancher server and then access a third-party site hosted by the exploiter. Once that is accomplished, the exploiter is able to execute commands against the Kubernetes API with the permissions and identity of the victim. Reported by Matt Belisle and Alex Stevenson from Workiva. | 15 Jul 2019 | [Rancher v2.2.5](https://github.com/rancher/rancher/releases/tag/v2.2.5), [Rancher v2.1.11](https://github.com/rancher/rancher/releases/tag/v2.1.11) and [Rancher v2.0.16](https://github.com/rancher/rancher/releases/tag/v2.0.16) |
| [CVE-2019-12303](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-12303) | Project owners can inject extra fluentd logging configurations that makes it possible to read files or execute arbitrary commands inside the fluentd container. Reported by Tyler Welton from Untamed Theory. | 5 Jun 2019 | [Rancher v2.2.4](https://github.com/rancher/rancher/releases/tag/v2.2.4), [Rancher v2.1.10](https://github.com/rancher/rancher/releases/tag/v2.1.10) and [Rancher v2.0.15](https://github.com/rancher/rancher/releases/tag/v2.0.15) |
| [CVE-2019-12274](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-12274) | Nodes using the built-in node drivers using a file path option allows the machine to read arbitrary files including sensitive ones from inside the Rancher server container. | 5 Jun 2019 | [Rancher v2.2.4](https://github.com/rancher/rancher/releases/tag/v2.2.4), [Rancher v2.1.10](https://github.com/rancher/rancher/releases/tag/v2.1.10) and [Rancher v2.0.15](https://github.com/rancher/rancher/releases/tag/v2.0.15) |
| [CVE-2019-11202](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-11202) | The default admin, that is shipped with Rancher, will be re-created upon restart of Rancher despite being explicitly deleted. | 16 Apr 2019 | [Rancher v2.2.2](https://github.com/rancher/rancher/releases/tag/v2.2.2), [Rancher v2.1.9](https://github.com/rancher/rancher/releases/tag/v2.1.9) and [Rancher v2.0.14](https://github.com/rancher/rancher/releases/tag/v2.0.14) |
| [CVE-2019-6287](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-6287) | Project members continue to get access to namespaces from projects that they were removed from if they were added to more than one project. | 29 Jan 2019 | [Rancher v2.1.6](https://github.com/rancher/rancher/releases/tag/v2.1.6) and [Rancher v2.0.11](https://github.com/rancher/rancher/releases/tag/v2.0.11) |
| [CVE-2018-20321](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-20321) | Any project member with access to the `default` namespace can mount the `netes-default` service account in a pod and then use that pod to execute administrative privileged commands against the Kubernetes cluster. | 29 Jan 2019 | [Rancher v2.1.6](https://github.com/rancher/rancher/releases/tag/v2.1.6) and [Rancher v2.0.11](https://github.com/rancher/rancher/releases/tag/v2.0.11) - Rolling back from these versions or greater have specific [instructions](../../getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/rollbacks.md). |
docs/reference-guides/rancher-security/selinux-rpm/about-rancher-selinux.md
+1
View File
@@ -0,0 +1 @@
<!-- PLACEHOLDER -->
docs/reference-guides/rancher-security/selinux-rpm/about-rke2-selinux.md
+1
View File
@@ -0,0 +1 @@
<!-- PLACEHOLDER -->
docs/reference-guides/rke1-template-example-yaml.md
+112
View File
@@ -0,0 +1,112 @@
---
title: Example YAML
weight: 60
---
Below is an example RKE template configuration file for reference.
The YAML in the RKE template uses the same customization that is used when you create an RKE cluster. However, since the YAML is within the context of a Rancher provisioned RKE cluster, the customization from the RKE docs needs to be nested under the `rancher_kubernetes_engine` directive.
```yaml
#
# Cluster Config
#
docker_root_dir: /var/lib/docker
enable_cluster_alerting: false
# This setting is not enforced. Clusters
# created with this sample template
# would have alerting turned off by default,
# but end users could still turn alerting
# on or off.
enable_cluster_monitoring: true
# This setting is not enforced. Clusters
# created with this sample template
# would have monitoring turned on
# by default, but end users could still
# turn monitoring on or off.
enable_network_policy: false
local_cluster_auth_endpoint:
enabled: true
#
# Rancher Config
#
rancher_kubernetes_engine_config: # Your RKE template config goes here.
addon_job_timeout: 30
authentication:
strategy: x509
ignore_docker_version: true
#
# # Currently only nginx ingress provider is supported.
# # To disable ingress controller, set `provider: none`
# # To enable ingress on specific nodes, use the node_selector, eg:
# provider: nginx
# node_selector:
# app: ingress
#
ingress:
provider: nginx
kubernetes_version: v1.15.3-rancher3-1
monitoring:
provider: metrics-server
#
# If you are using calico on AWS
#
# network:
# plugin: calico
# calico_network_provider:
# cloud_provider: aws
#
# # To specify flannel interface
#
# network:
# plugin: flannel
# flannel_network_provider:
# iface: eth1
#
# # To specify flannel interface for canal plugin
#
# network:
# plugin: canal
# canal_network_provider:
# iface: eth1
#
network:
options:
flannel_backend_type: vxlan
plugin: canal
#
# services:
# kube-api:
# service_cluster_ip_range: 10.43.0.0/16
# kube-controller:
# cluster_cidr: 10.42.0.0/16
# service_cluster_ip_range: 10.43.0.0/16
# kubelet:
# cluster_domain: cluster.local
# cluster_dns_server: 10.43.0.10
#
services:
etcd:
backup_config:
enabled: true
interval_hours: 12
retention: 6
safe_timestamp: false
creation: 12h
extra_args:
election-timeout: 5000
heartbeat-interval: 500
gid: 0
retention: 72h
snapshot: false
uid: 0
kube_api:
always_pull_images: false
pod_security_policy: false
service_node_port_range: 30000-32767
ssh_agent_auth: false
windows_prefered_cluster: false
```
docs/reference-guides/single-node-rancher-in-docker/advanced-options.md
+115
View File
@@ -0,0 +1,115 @@
---
title: Advanced Options for Docker Installs
weight: 5
---
When installing Rancher, there are several advanced options that can be enabled:
- [Custom CA Certificate](#custom-ca-certificate)
- [API Audit Log](#api-audit-log)
- [TLS Settings](#tls-settings)
- [Air Gap](#air-gap)
- [Persistent Data](#persistent-data)
- [Running `rancher/rancher` and `rancher/rancher-agent` on the Same Node](#running-rancher-rancher-and-rancher-rancher-agent-on-the-same-node)
### Custom CA Certificate
If you want to configure Rancher to use a CA root certificate to be used when validating services, you would start the Rancher container sharing the directory that contains the CA root certificate.
Use the command example to start a Rancher container with your private CA certificates mounted.
- The volume flag (`-v`) should specify the host directory containing the CA root certificates.
- The environment variable flag (`-e`) in combination with `SSL_CERT_DIR` and directory declares an environment variable that specifies the mounted CA root certificates directory location inside the container.
- Passing environment variables to the Rancher container can be done using `-e KEY=VALUE` or `--env KEY=VALUE`.
- Mounting a host directory inside the container can be done using `-v host-source-directory:container-destination-directory` or `--volume host-source-directory:container-destination-directory`.
The example below is based on having the CA root certificates in the `/host/certs` directory on the host and mounting this directory on `/container/certs` inside the Rancher container.
Privileged access is [required.](../../pages-for-subheaders/rancher-on-a-single-node-with-docker.md#privileged-access-for-rancher)
```
docker run -d --restart=unless-stopped \
-p 80:80 -p 443:443 \
-v /host/certs:/container/certs \
-e SSL_CERT_DIR="/container/certs" \
--privileged \
rancher/rancher:latest
```
### API Audit Log
The API Audit Log records all the user and system transactions made through Rancher server.
The API Audit Log writes to `/var/log/auditlog` inside the rancher container by default. Share that directory as a volume and set your `AUDIT_LEVEL` to enable the log.
See [API Audit Log](../../getting-started/installation-and-upgrade/advanced-options/advanced-use-cases/enable-api-audit-log.md) for more information and options.
Privileged access is [required.](../../pages-for-subheaders/rancher-on-a-single-node-with-docker.md#privileged-access-for-rancher)
```
docker run -d --restart=unless-stopped \
-p 80:80 -p 443:443 \
-v /var/log/rancher/auditlog:/var/log/auditlog \
-e AUDIT_LEVEL=1 \
--privileged \
rancher/rancher:latest
```
### TLS settings
To set a different TLS configuration, you can use the `CATTLE_TLS_MIN_VERSION` and `CATTLE_TLS_CIPHERS` environment variables. For example, to configure TLS 1.0 as minimum accepted TLS version:
```
docker run -d --restart=unless-stopped \
-p 80:80 -p 443:443 \
-e CATTLE_TLS_MIN_VERSION="1.0" \
--privileged \
rancher/rancher:latest
```
Privileged access is [required.](../../pages-for-subheaders/rancher-on-a-single-node-with-docker.md#privileged-access-for-rancher)
See [TLS settings](../installation-references/tls-settings.md) for more information and options.
### Air Gap
If you are visiting this page to complete an air gap installation, you must prepend your private registry URL to the server tag when running the installation command in the option that you choose. Add `<REGISTRY.DOMAIN.COM:PORT>` with your private registry URL in front of `rancher/rancher:latest`.
**Example:**
<REGISTRY.DOMAIN.COM:PORT>/rancher/rancher:latest
### Persistent Data
Rancher uses etcd as a datastore. When Rancher is installed with Docker, the embedded etcd is being used. The persistent data is at the following path in the container: `/var/lib/rancher`.
You can bind mount a host volume to this location to preserve data on the host it is running on:
```
docker run -d --restart=unless-stopped \
-p 80:80 -p 443:443 \
-v /opt/rancher:/var/lib/rancher \
--privileged \
rancher/rancher:latest
```
Privileged access is [required.](../../pages-for-subheaders/rancher-on-a-single-node-with-docker.md#privileged-access-for-rancher)
### Running `rancher/rancher` and `rancher/rancher-agent` on the Same Node
In the situation where you want to use a single node to run Rancher and to be able to add the same node to a cluster, you have to adjust the host ports mapped for the `rancher/rancher` container.
If a node is added to a cluster, it deploys the nginx ingress controller which will use port 80 and 443. This will conflict with the default ports we advise to expose for the `rancher/rancher` container.
Please note that this setup is not recommended for production use, but can be convenient for development/demo purposes.
To change the host ports mapping, replace the following part `-p 80:80 -p 443:443` with `-p 8080:80 -p 8443:443`:
```
docker run -d --restart=unless-stopped \
-p 8080:80 -p 8443:443 \
--privileged \
rancher/rancher:latest
```
Privileged access is [required.](../../pages-for-subheaders/rancher-on-a-single-node-with-docker.md#privileged-access-for-rancher)
docs/reference-guides/single-node-rancher-in-docker/http-proxy-configuration.md
+64
View File
@@ -0,0 +1,64 @@
---
title: HTTP Proxy Configuration
weight: 251
---
If you operate Rancher behind a proxy and you want to access services through the proxy (such as retrieving catalogs), you must provide Rancher information about your proxy. As Rancher is written in Go, it uses the common proxy environment variables as shown below.
Make sure `NO_PROXY` contains the network addresses, network address ranges and domains that should be excluded from using the proxy.
| Environment variable | Purpose |
| -------------------- | ----------------------------------------------------------------------------------------------------------------------- |
| HTTP_PROXY | Proxy address to use when initiating HTTP connection(s) |
| HTTPS_PROXY | Proxy address to use when initiating HTTPS connection(s) |
| NO_PROXY | Network address(es), network address range(s) and domains to exclude from using the proxy when initiating connection(s) |
:::note Important:
NO_PROXY must be in uppercase to use network range (CIDR) notation.
:::
## Docker Installation
Passing environment variables to the Rancher container can be done using `-e KEY=VALUE` or `--env KEY=VALUE`. Required values for `NO_PROXY` in a [Docker Installation](../../pages-for-subheaders/rancher-on-a-single-node-with-docker.md) are:
- `localhost`
- `127.0.0.1`
- `0.0.0.0`
- `10.0.0.0/8`
- `cattle-system.svc`
- `.svc`
- `.cluster.local`
The example below is based on a proxy server accessible at `http://192.168.0.1:3128`, and excluding usage the proxy when accessing network range `192.168.10.0/24` and every hostname under the domain `example.com`.
```
docker run -d --restart=unless-stopped \
-p 80:80 -p 443:443 \
-e HTTP_PROXY="http://192.168.10.1:3128" \
-e HTTPS_PROXY="http://192.168.10.1:3128" \
-e NO_PROXY="localhost,127.0.0.1,0.0.0.0,10.0.0.0/8,cattle-system.svc,192.168.10.0/24,.svc,.cluster.local,example.com" \
--privileged \
rancher/rancher:latest
```
Privileged access is [required.](../../pages-for-subheaders/rancher-on-a-single-node-with-docker.md#privileged-access-for-rancher)
### Air-gapped proxy configuration
_New in v2.6.4_
You can now provision node driver clusters from an air-gapped cluster configured to use a proxy for outbound connections.
In addition to setting the default rules for a proxy server as shown above, you will need to add additional rules, shown below, to provision node driver clusters from a proxied Rancher environment.
You will configure your filepath according to your setup, e.g., `/etc/apt/apt.conf.d/proxy.conf`:
```
acl SSL_ports port 22
acl SSL_ports port 2376
acl Safe_ports port 22 # ssh
acl Safe_ports port 2376 # docker port
```
docs/reference-guides/system-tools.md
+31
View File
@@ -0,0 +1,31 @@
---
title: System Tools
weight: 22
---
:::note
System Tools has been deprecated since June 2022.
:::
# Logs
Please use [logs-collector](https://github.com/rancherlabs/support-tools/tree/master/collection/rancher/v2.x/logs-collector) to collect logs from your cluster.
# Stats
If you want to replicate the stats command, you can run the following command on your cluster nodes:
:::note
This command below requires the package `sysstat` on the cluster node.
:::
```
/usr/bin/sar -u -r -F 1 1
```
# Remove
Please use the [Rancher Cleanup](https://github.com/rancher/rancher-cleanup) tool.
docs/reference-guides/user-settings/api-keys.md
+60
View File
@@ -0,0 +1,60 @@
---
title: API Keys
weight: 7005
---
## API Keys and User Authentication
If you want to access your Rancher clusters, projects, or other objects using external applications, you can do so using the Rancher API. However, before your application can access the API, you must provide the app with a key used to authenticate with Rancher. You can obtain a key using the Rancher UI.
An API key is also required for using Rancher CLI.
API Keys are composed of four components:
- **Endpoint:** This is the IP address and path that other applications use to send requests to the Rancher API.
- **Access Key:** The token's username.
- **Secret Key:** The token's password. For applications that prompt you for two different strings for API authentication, you usually enter the two keys together.
- **Bearer Token:** The token username and password concatenated together. Use this string for applications that prompt you for one authentication string.
:::note
Users may opt to enable [token hashing](../about-the-api/api-tokens.md).
:::
## Creating an API Key
1. Select **User Avatar > Account & API Keys** from upper right corner.
2. Click **Create API Key**.
3. **Optional:** Enter a description for the API key and select an expiration period or a scope. We recommend setting an expiration date.
The API key won't be valid after expiration. Shorter expiration periods are more secure.
Expiration period will be bound by `v3/settings/auth-token-max-ttl-minutes`. If it exceeds the max-ttl, API key will be created with max-ttl as the expiration period.
A scope will limit the API key so that it will only work against the Kubernetes API of the specified cluster. If the cluster is configured with an Authorized Cluster Endpoint, you will be able to use a scoped token directly against the cluster's API without proxying through the Rancher server. See [Authorized Cluster Endpoints](../../pages-for-subheaders/rancher-manager-architecture.md#4-authorized-cluster-endpoint) for more information.
4. Click **Create**.
**Step Result:** Your API Key is created. Your API **Endpoint**, **Access Key**, **Secret Key**, and **Bearer Token** are displayed.
Use the **Bearer Token** to authenticate with Rancher CLI.
5. Copy the information displayed to a secure location. This information is only displayed once, so if you lose your key, you'll have to make a new one.
## What's Next?
- Enter your API key information into the application that will send requests to the Rancher API.
- Learn more about the Rancher endpoints and parameters by selecting **View in API** for an object in the Rancher UI.
- API keys are used for API calls and [Rancher CLI](../../pages-for-subheaders/cli-with-rancher.md).
## Deleting API Keys
If you need to revoke an API key, delete it. You should delete API keys:
- That may have been compromised.
- That have expired.
To delete an API, select the stale key and click **Delete**.
docs/reference-guides/user-settings/manage-cloud-credentials.md
+52
View File
@@ -0,0 +1,52 @@
---
title: Managing Cloud Credentials
weight: 7011
---
When you create a cluster [hosted by an infrastructure provider](../../pages-for-subheaders/use-new-nodes-in-an-infra-provider.md), [node templates](../../pages-for-subheaders/use-new-nodes-in-an-infra-provider.md#node-templates) are used to provision the cluster nodes. These templates use Docker Machine configuration options to define an operating system image and settings/parameters for the node.
Node templates can use cloud credentials to access the credential information required to provision nodes in the infrastructure providers. The same cloud credential can be used by multiple node templates. By using a cloud credential, you do not have to re-enter access keys for the same cloud provider. Cloud credentials are stored as Kubernetes secrets.
Cloud credentials are only used by node templates if there are fields marked as `password`. The default `active` node drivers have their account access fields marked as `password`, but there may be some `inactive` node drivers, which are not using them yet. These node drivers will not use cloud credentials.
You can create cloud credentials in two contexts:
- [During creation of a node template](../../pages-for-subheaders/use-new-nodes-in-an-infra-provider.md#node-templates) for a cluster.
- In the **User Settings**
All cloud credentials are bound to the user profile of who created it. They **cannot** be shared across users.
## Creating a Cloud Credential from User Settings
1. Click **☰ > Cluster Management**.
1. Click **Cloud Credentials**.
1. Click **Create**.
1. Click a cloud credential type. The values of this dropdown is based on the `active` [node drivers](../../how-to-guides/advanced-user-guides/authentication-permissions-and-global-configuration/about-provisioning-drivers/manage-node-drivers.md) in Rancher.
1. Enter a name for the cloud credential.
1. Based on the selected cloud credential type, enter the required values to authenticate with the infrastructure provider.
1. Click **Create**.
**Result:** The cloud credential is created and can immediately be used to [create node templates](../../pages-for-subheaders/use-new-nodes-in-an-infra-provider.md#node-templates).
## Updating a Cloud Credential
When access credentials are changed or compromised, updating a cloud credential allows you to rotate those credentials while keeping the same node template.
1. Click **☰ > Cluster Management**.
1. Click **Cloud Credentials**.
1. Choose the cloud credential you want to edit and click the **⋮ > Edit Config**.
1. Update the credential information and click **Save**.
**Result:** The cloud credential is updated with the new access credentials. All existing node templates using this cloud credential will automatically use the updated information whenever [new nodes are added](../../pages-for-subheaders/use-new-nodes-in-an-infra-provider.md).
## Deleting a Cloud Credential
In order to delete cloud credentials, there must not be any node template associated with it. If you are unable to delete the cloud credential, [delete any node templates](manage-node-templates.md#deleting-a-node-template) that are still associated to that cloud credential.
1. Click **☰ > Cluster Management**.
1. Click **Cloud Credentials**.
1. You can either individually delete a cloud credential or bulk delete.
- To individually delete one, choose the cloud credential you want to edit and click the **⋮ > Delete**.
- To bulk delete cloud credentials, select one or more cloud credentials from the list. Click **Delete**.
1. Confirm that you want to delete these cloud credentials.
docs/reference-guides/user-settings/manage-node-templates.md
+55
View File
@@ -0,0 +1,55 @@
---
title: Managing Node Templates
weight: 7010
---
When you provision a cluster [hosted by an infrastructure provider](../../pages-for-subheaders/use-new-nodes-in-an-infra-provider.md), [node templates](../../pages-for-subheaders/use-new-nodes-in-an-infra-provider.md#node-templates) are used to provision the cluster nodes. These templates use Docker Machine configuration options to define an operating system image and settings/parameters for the node. You can create node templates in two contexts:
- While [provisioning a node pool cluster](../../pages-for-subheaders/use-new-nodes-in-an-infra-provider.md).
- At any time, from your [user settings](#creating-a-node-template-from-user-settings).
When you create a node template, it is bound to your user profile. Node templates cannot be shared among users. You can delete stale node templates that you no longer user from your user settings.
## Creating a Node Template
1. Click **☰ > Cluster Management**.
1. Click **RKE1 Configuration > Node Templates**.
1. Click **Add Template**.
1. Select one of the cloud providers available. Then follow the instructions on screen to configure the template.
**Result:** The template is configured. You can use the template later when you [provision a node pool cluster](../../pages-for-subheaders/use-new-nodes-in-an-infra-provider.md).
## Updating a Node Template
1. Click **☰ > Cluster Management**.
1. Click **RKE1 Configuration > Node Templates**.
1. Choose the node template that you want to edit and click the **⋮ > Edit**.
:::note
The default `active` [node drivers](../../how-to-guides/advanced-user-guides/authentication-permissions-and-global-configuration/about-provisioning-drivers/manage-node-drivers.md) and any node driver, that has fields marked as `password`, are required to use [cloud credentials](../../pages-for-subheaders/use-new-nodes-in-an-infra-provider.md#cloud-credentials).
:::
1. Edit the required information and click **Save**.
**Result:** The node template is updated. All node pools using this node template will automatically use the updated information when new nodes are added.
## Cloning Node Templates
When creating new node templates from your user settings, you can clone an existing template and quickly update its settings rather than creating a new one from scratch. Cloning templates saves you the hassle of re-entering access keys for the cloud provider.
1. Click **☰ > Cluster Management**.
1. Click **RKE1 Configuration > Node Templates**.
1. Find the template you want to clone. Then select **⋮ > Clone**.
1. Complete the rest of the form.
**Result:** The template is cloned and configured. You can use the template later when you [provision a node pool cluster](../../pages-for-subheaders/use-new-nodes-in-an-infra-provider.md).
## Deleting a Node Template
When you no longer use a node template, you can delete it from your user settings.
1. Click **☰ > Cluster Management**.
1. Click **RKE1 Configuration > Node Templates**.
1. Select one or more template from the list. Then click **Delete**. Confirm the delete when prompted.
docs/reference-guides/user-settings/user-preferences.md
+18
View File
@@ -0,0 +1,18 @@
---
title: User Preferences
weight: 7012
---
Each user can choose preferences to personalize their Rancher experience. To change preference settings, open the **User Settings** menu and then select **Preferences**.
## Theme
Choose your background color for the Rancher UI. If you choose **Auto**, the background color changes from light to dark at 6 PM, and then changes back at 6 AM.
## My Account
This section displays the **Name** (your display name) and **Username** (your login) used for your session. To change your login's current password, click the **Change Password** button.
## Table Row per Page
On pages that display system objects like clusters or deployments in a table, you can set the number of objects that display on the page before you must paginate. The default setting is `50`.